Note: This is test shard 2 of 4.
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from HmsConfigurations/MasterFailoverTest
[ RUN ] HmsConfigurations/MasterFailoverTest.TestCreateTableSync/1
2025-09-04T01:57:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T01:57:13Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
2025-09-04 01:57:18: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250904 01:57:26.714311 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:43223
--webserver_interface=127.16.112.190
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--hive_metastore_uris=thrift://127.0.0.1:35453
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
W20250904 01:57:26.995772 17077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:26.996330 17077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:26.996798 17077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:27.025950 17077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:57:27.026257 17077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:27.026520 17077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:57:27.026753 17077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:57:27.058983 17077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35453
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.112.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:27.060230 17077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:27.061777 17077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:27.071658 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:27.072180 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:28.221949 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:28.225921 17085 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1152 milliseconds
W20250904 01:57:28.226490 17077 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.155s user 0.353s sys 0.796s
W20250904 01:57:28.226754 17077 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.155s user 0.353s sys 0.796s
I20250904 01:57:28.226980 17077 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:57:28.227994 17077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:28.231753 17077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:28.233075 17077 hybrid_clock.cc:648] HybridClock initialized: now 1756951048233028 us; error 55 us; skew 500 ppm
I20250904 01:57:28.233860 17077 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:28.240620 17077 webserver.cc:480] Webserver started at http://127.16.112.190:35179/ using document root <none> and password file <none>
I20250904 01:57:28.241464 17077 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:28.241703 17077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:28.242117 17077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:28.246388 17077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/instance:
uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
format_stamp: "Formatted at 2025-09-04 01:57:28 on dist-test-slave-pl7p"
I20250904 01:57:28.247453 17077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance:
uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
format_stamp: "Formatted at 2025-09-04 01:57:28 on dist-test-slave-pl7p"
I20250904 01:57:28.254900 17077 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250904 01:57:28.260591 17093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:28.261643 17077 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250904 01:57:28.261932 17077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
format_stamp: "Formatted at 2025-09-04 01:57:28 on dist-test-slave-pl7p"
I20250904 01:57:28.262240 17077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:28.350121 17077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:28.351475 17077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:28.351886 17077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:28.415222 17077 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.190:43223
I20250904 01:57:28.415284 17144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.190:43223 every 8 connection(s)
I20250904 01:57:28.417817 17077 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
I20250904 01:57:28.422346 17145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:57:28.426553 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17077
I20250904 01:57:28.426959 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance
I20250904 01:57:28.430358 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:39353
--webserver_interface=127.16.112.189
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--hive_metastore_uris=thrift://127.0.0.1:35453
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:57:28.432346 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:28.461383 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:57:28.463546 17095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
W20250904 01:57:28.467237 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
I20250904 01:57:28.516747 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 1
W20250904 01:57:28.520308 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
I20250904 01:57:28.588728 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 2
W20250904 01:57:28.592284 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
I20250904 01:57:28.683732 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 3
W20250904 01:57:28.687453 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
W20250904 01:57:28.707746 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:28.708299 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:28.708767 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:28.737059 17146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:57:28.737352 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:28.737624 17146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:57:28.737869 17146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:57:28.770161 17146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:35453
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.112.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:28.771399 17146 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:28.772845 17146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:28.782536 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:28.830854 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 4
W20250904 01:57:28.834733 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
I20250904 01:57:29.134022 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 5
W20250904 01:57:29.137455 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
I20250904 01:57:29.685068 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 6
W20250904 01:57:29.692202 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:39353: Network error: Client connection negotiation failed: client connection to 127.16.112.189:39353: connect: Connection refused (error 111)
W20250904 01:57:28.783327 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:28.786923 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:29.901036 17157 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250904 01:57:29.901144 17146 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:57:29.905215 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:29.908290 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:29.909660 17146 hybrid_clock.cc:648] HybridClock initialized: now 1756951049909625 us; error 54 us; skew 500 ppm
I20250904 01:57:29.910390 17146 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:29.916608 17146 webserver.cc:480] Webserver started at http://127.16.112.189:35101/ using document root <none> and password file <none>
I20250904 01:57:29.917485 17146 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:29.917726 17146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:29.918149 17146 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:29.922796 17146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/instance:
uuid: "e73ac0fbf752482195cd3a0207ae6eba"
format_stamp: "Formatted at 2025-09-04 01:57:29 on dist-test-slave-pl7p"
I20250904 01:57:29.923776 17146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance:
uuid: "e73ac0fbf752482195cd3a0207ae6eba"
format_stamp: "Formatted at 2025-09-04 01:57:29 on dist-test-slave-pl7p"
I20250904 01:57:29.930706 17146 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250904 01:57:29.936479 17167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:29.937407 17146 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250904 01:57:29.937747 17146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
uuid: "e73ac0fbf752482195cd3a0207ae6eba"
format_stamp: "Formatted at 2025-09-04 01:57:29 on dist-test-slave-pl7p"
I20250904 01:57:29.938046 17146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:29.984468 17146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:29.986290 17146 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:29.986816 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:30.054478 17146 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.189:39353
I20250904 01:57:30.054529 17218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.189:39353 every 8 connection(s)
I20250904 01:57:30.057148 17146 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
I20250904 01:57:30.061721 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:57:30.066928 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17146
I20250904 01:57:30.067415 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance
I20250904 01:57:30.070539 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:43009
--webserver_interface=127.16.112.188
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--hive_metastore_uris=thrift://127.0.0.1:35453
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:57:30.075420 17219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:30.094005 17219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:30.106277 17219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:57:30.108314 17169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
W20250904 01:57:30.111697 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.161178 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 1
W20250904 01:57:30.164726 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.233160 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 2
W20250904 01:57:30.237357 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.328758 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 3
W20250904 01:57:30.332593 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
W20250904 01:57:30.369937 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:30.370507 17220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:30.370954 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:30.400908 17220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:57:30.401242 17220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:30.401484 17220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:57:30.401746 17220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:57:30.444615 17220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35453
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.112.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:30.445982 17220 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:30.447516 17220 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:30.458797 17230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:30.475944 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 4
W20250904 01:57:30.482483 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.752777 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } attempt: 7
I20250904 01:57:30.781975 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 5
W20250904 01:57:30.788745 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.797475 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:57:30.807435 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.865931 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 1
W20250904 01:57:30.872345 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:30.953827 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 2
W20250904 01:57:30.960337 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:31.045859 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 3
W20250904 01:57:31.053738 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:31.194278 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 4
W20250904 01:57:31.201630 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:31.336315 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 6
W20250904 01:57:31.385353 17219 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
I20250904 01:57:31.485205 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 5
W20250904 01:57:31.493127 17145 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:43009: Network error: Client connection negotiation failed: client connection to 127.16.112.188:43009: connect: Connection refused (error 111)
W20250904 01:57:30.459473 17231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:31.793632 17233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:31.796448 17232 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1331 milliseconds
W20250904 01:57:31.796435 17220 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.337s user 0.454s sys 0.877s
W20250904 01:57:31.796880 17220 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.338s user 0.454s sys 0.877s
I20250904 01:57:31.797183 17220 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:57:31.798832 17220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:31.802019 17220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:31.803516 17220 hybrid_clock.cc:648] HybridClock initialized: now 1756951051803476 us; error 33 us; skew 500 ppm
I20250904 01:57:31.804633 17220 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:31.813475 17220 webserver.cc:480] Webserver started at http://127.16.112.188:34163/ using document root <none> and password file <none>
I20250904 01:57:31.814926 17220 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:31.815201 17220 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:31.815775 17220 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:31.891253 17220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/instance:
uuid: "616c90f4d4be4e11bb7b577e6528a449"
format_stamp: "Formatted at 2025-09-04 01:57:31 on dist-test-slave-pl7p"
I20250904 01:57:31.892360 17220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance:
uuid: "616c90f4d4be4e11bb7b577e6528a449"
format_stamp: "Formatted at 2025-09-04 01:57:31 on dist-test-slave-pl7p"
I20250904 01:57:31.899668 17220 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.009s
I20250904 01:57:31.904847 17245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:31.905805 17220 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250904 01:57:31.906071 17220 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
uuid: "616c90f4d4be4e11bb7b577e6528a449"
format_stamp: "Formatted at 2025-09-04 01:57:31 on dist-test-slave-pl7p"
I20250904 01:57:31.906337 17220 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:31.980026 17220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:31.981418 17220 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:31.981833 17220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:32.045660 17145 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 6
I20250904 01:57:32.051936 17220 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.188:43009
I20250904 01:57:32.052006 17298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.188:43009 every 8 connection(s)
I20250904 01:57:32.056295 17220 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
I20250904 01:57:32.061038 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17220
I20250904 01:57:32.061556 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance
I20250904 01:57:32.062315 17300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:57:32.080144 17300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:32.091996 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: Bootstrap starting.
I20250904 01:57:32.103752 17145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: Neither blocks nor log segments found. Creating new log.
I20250904 01:57:32.105957 17145 log.cc:826] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: Log is configured to *not* fsync() on all Append() calls
I20250904 01:57:32.112349 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: No bootstrap required, opened a new log
I20250904 01:57:32.116988 17300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:32.132351 17300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:32.135149 17145 raft_consensus.cc:357] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.135936 17145 raft_consensus.cc:383] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:57:32.136165 17145 raft_consensus.cc:738] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bba492ab9e479ba4a4ec6e37ece375, State: Initialized, Role: FOLLOWER
I20250904 01:57:32.136893 17145 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.141753 17309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.142621 17309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.143451 17145 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:57:32.149406 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Bootstrap starting.
I20250904 01:57:32.155755 17300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Neither blocks nor log segments found. Creating new log.
I20250904 01:57:32.157545 17300 log.cc:826] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Log is configured to *not* fsync() on all Append() calls
I20250904 01:57:32.165112 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: No bootstrap required, opened a new log
I20250904 01:57:32.190068 17300 raft_consensus.cc:357] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.190912 17300 raft_consensus.cc:383] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:57:32.191210 17300 raft_consensus.cc:738] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616c90f4d4be4e11bb7b577e6528a449, State: Initialized, Role: FOLLOWER
I20250904 01:57:32.192170 17300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.195432 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.196146 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.198136 17300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: configured and running, proceeding with master startup.
W20250904 01:57:32.235314 17325 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:57:32.235647 17325 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250904 01:57:32.245929 17251 tablet.cc:2378] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:57:32.201522 17077 master_runner.cc:186] Error getting master registration for 127.16.112.189:39353: OK, instance_id { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" instance_seqno: 1756951050035880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:57:32.248214 17077 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:57:32.301211 17342 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250904 01:57:32.317951 17309 raft_consensus.cc:491] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:57:32.318516 17309 raft_consensus.cc:513] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.320900 17309 leader_election.cc:290] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:32.322497 17272 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:32.323315 17272 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8bba492ab9e479ba4a4ec6e37ece375 in term 0.
I20250904 01:57:32.324502 17095 leader_election.cc:304] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375; no voters:
I20250904 01:57:32.325239 17309 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:57:32.325580 17309 raft_consensus.cc:491] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:57:32.325902 17309 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:57:32.313318 17220 master_runner.cc:186] Error getting master registration for 127.16.112.189:39353: OK, instance_id { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" instance_seqno: 1756951050035880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:57:32.351538 17220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:57:32.359917 17099 tablet.cc:2378] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:57:32.400218 17309 raft_consensus.cc:513] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.403748 17272 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449"
I20250904 01:57:32.404317 17272 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:57:32.409794 17309 leader_election.cc:290] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [CANDIDATE]: Term 1 election: Requested vote from peers e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:32.411468 17272 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8bba492ab9e479ba4a4ec6e37ece375 in term 1.
I20250904 01:57:32.412354 17095 leader_election.cc:304] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375; no voters:
I20250904 01:57:32.413029 17309 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:57:32.421649 17309 raft_consensus.cc:695] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 LEADER]: Becoming Leader. State: Replica: d8bba492ab9e479ba4a4ec6e37ece375, State: Running, Role: LEADER
I20250904 01:57:32.422539 17309 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.432816 17343 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bba492ab9e479ba4a4ec6e37ece375. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.433581 17343 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:32.437611 17345 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:57:32.445628 17345 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:57:32.445950 17219 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } attempt: 7
I20250904 01:57:32.479019 17272 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Refusing update from remote peer d8bba492ab9e479ba4a4ec6e37ece375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:57:32.480839 17343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:57:32.495306 17219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: Bootstrap starting.
I20250904 01:57:32.502592 17219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: Neither blocks nor log segments found. Creating new log.
I20250904 01:57:32.504688 17219 log.cc:826] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: Log is configured to *not* fsync() on all Append() calls
I20250904 01:57:32.509941 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bba492ab9e479ba4a4ec6e37ece375. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.510581 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.516127 17219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: No bootstrap required, opened a new log
I20250904 01:57:32.526926 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.527608 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.522805 17348 mvcc.cc:204] Tried to move back new op lower bound from 7196471510934642688 to 7196471510729515008. Current Snapshot: MvccSnapshot[applied={T|T < 7196471510934642688}]
I20250904 01:57:32.535488 17343 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.536193 17343 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:32.540324 17345 catalog_manager.cc:1349] Generated new cluster ID: ff74342adf9a48c0ac0b7771a1ce3900
I20250904 01:57:32.540611 17345 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:57:32.543385 17219 raft_consensus.cc:357] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.544153 17219 raft_consensus.cc:383] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:57:32.544415 17219 raft_consensus.cc:738] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73ac0fbf752482195cd3a0207ae6eba, State: Initialized, Role: FOLLOWER
I20250904 01:57:32.545230 17219 consensus_queue.cc:260] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:32.572472 17219 sys_catalog.cc:564] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:57:32.572649 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba"
I20250904 01:57:32.573315 17193 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:57:32.571151 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.576936 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.579686 17194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:32.583436 17193 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8bba492ab9e479ba4a4ec6e37ece375 in term 1.
I20250904 01:57:32.584918 17192 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Refusing update from remote peer d8bba492ab9e479ba4a4ec6e37ece375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:57:32.596139 17343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:57:32.602797 17345 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 01:57:32.604621 17345 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:57:32.633493 17345 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: Generated new TSK 0
I20250904 01:57:32.634433 17345 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:57:32.638883 17345 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:57:32.642620 17345 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:57:32.650707 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bba492ab9e479ba4a4ec6e37ece375. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.650753 17309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.651396 17309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:32.651397 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.651569 17359 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:32.652228 17359 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:32.664217 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.129:0
--local_ip_for_outbound_sockets=127.16.112.129
--webserver_interface=127.16.112.129
--webserver_port=0
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:57:32.713706 17374 catalog_manager.cc:1261] Loaded cluster ID: ff74342adf9a48c0ac0b7771a1ce3900
I20250904 01:57:32.713980 17374 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: loading cluster ID for follower catalog manager: success
I20250904 01:57:32.723152 17374 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: acquiring CA information for follower catalog manager: success
I20250904 01:57:32.727953 17374 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 01:57:33.097694 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:33.098397 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:33.099395 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:33.155506 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:33.157104 17367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.129
I20250904 01:57:33.170168 17380 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:33.170678 17380 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:33.172953 17380 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:33.174036 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:33.174299 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:33.176059 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:33.176780 17380 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:33.224162 17367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.112.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.129
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:33.226006 17367 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:33.228154 17367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:33.244518 17383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:33.326406 17342 catalog_manager.cc:1261] Loaded cluster ID: ff74342adf9a48c0ac0b7771a1ce3900
I20250904 01:57:33.326778 17342 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: loading cluster ID for follower catalog manager: success
I20250904 01:57:33.333940 17342 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: acquiring CA information for follower catalog manager: success
I20250904 01:57:33.345435 17342 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:57:33.553975 17389 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:33.555171 17389 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:33.560237 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:33.566910 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:33.568341 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:33.578950 17390 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:33.579656 17389 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:34.144181 17392 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:34.147205 17392 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:34.152281 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:34.162951 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:34.164407 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:34.174899 17393 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:34.180943 17392 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:35.078788 17394 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:35.079533 17394 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:35.082082 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:35.082576 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:35.084581 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:35.086285 17394 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:35.087519 17394 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:35.206789 17396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:35.213094 17396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:35.215543 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:35.218168 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:35.219736 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:35.226248 17396 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:35.227602 17397 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:36.195277 17398 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:36.197198 17398 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:36.201000 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:36.203184 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:36.204535 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:36.210407 17399 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:36.210426 17398 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
W20250904 01:57:34.645545 17381 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 17367
W20250904 01:57:36.239957 17384 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2994 milliseconds
W20250904 01:57:36.240414 17386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:36.239362 17367 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.994s user 0.825s sys 1.480s
W20250904 01:57:33.244928 17382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:36.241230 17367 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.996s user 0.826s sys 1.481s
I20250904 01:57:36.241523 17367 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:57:36.245457 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:36.248044 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:36.249517 17367 hybrid_clock.cc:648] HybridClock initialized: now 1756951056249403 us; error 113 us; skew 500 ppm
I20250904 01:57:36.250566 17367 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:36.258085 17367 webserver.cc:480] Webserver started at http://127.16.112.129:40671/ using document root <none> and password file <none>
I20250904 01:57:36.259353 17367 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:36.259678 17367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:36.260265 17367 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:36.266350 17367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/instance:
uuid: "3b1acf6128194f9c9e264688369d06f6"
format_stamp: "Formatted at 2025-09-04 01:57:36 on dist-test-slave-pl7p"
I20250904 01:57:36.267791 17367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance:
uuid: "3b1acf6128194f9c9e264688369d06f6"
format_stamp: "Formatted at 2025-09-04 01:57:36 on dist-test-slave-pl7p"
I20250904 01:57:36.276664 17367 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250904 01:57:36.283936 17404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:36.285038 17367 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250904 01:57:36.285434 17367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
uuid: "3b1acf6128194f9c9e264688369d06f6"
format_stamp: "Formatted at 2025-09-04 01:57:36 on dist-test-slave-pl7p"
I20250904 01:57:36.285946 17367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:36.357554 17367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:36.359558 17367 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:36.360131 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:36.364243 17367 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:57:36.370844 17367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:57:36.371150 17367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:36.371524 17367 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:57:36.371791 17367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:36.584669 17367 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.129:40831
I20250904 01:57:36.584790 17516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.129:40831 every 8 connection(s)
I20250904 01:57:36.588306 17367 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
I20250904 01:57:36.596863 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17367
I20250904 01:57:36.597460 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance
I20250904 01:57:36.615799 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.130:0
--local_ip_for_outbound_sockets=127.16.112.130
--webserver_interface=127.16.112.130
--webserver_port=0
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:57:36.653347 17517 heartbeater.cc:344] Connected to a master server at 127.16.112.188:43009
I20250904 01:57:36.653904 17517 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:36.655508 17517 heartbeater.cc:507] Master 127.16.112.188:43009 requested a full tablet report, sending...
I20250904 01:57:36.659423 17262 ts_manager.cc:194] Registered new tserver with Master: 3b1acf6128194f9c9e264688369d06f6 (127.16.112.129:40831)
I20250904 01:57:36.663224 17518 heartbeater.cc:344] Connected to a master server at 127.16.112.189:39353
I20250904 01:57:36.663632 17518 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:36.664664 17518 heartbeater.cc:507] Master 127.16.112.189:39353 requested a full tablet report, sending...
I20250904 01:57:36.667825 17184 ts_manager.cc:194] Registered new tserver with Master: 3b1acf6128194f9c9e264688369d06f6 (127.16.112.129:40831)
I20250904 01:57:36.683007 17521 heartbeater.cc:344] Connected to a master server at 127.16.112.190:43223
I20250904 01:57:36.683350 17521 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:36.684230 17521 heartbeater.cc:507] Master 127.16.112.190:43223 requested a full tablet report, sending...
I20250904 01:57:36.686576 17110 ts_manager.cc:194] Registered new tserver with Master: 3b1acf6128194f9c9e264688369d06f6 (127.16.112.129:40831)
I20250904 01:57:36.689044 17110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.129:56983
I20250904 01:57:36.785960 17527 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:36.786643 17527 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:36.789242 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:36.789748 17527 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:36.790673 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:36.791908 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:36.792790 17527 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:57:36.968791 17525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:36.969277 17525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:36.969935 17525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:37.007495 17525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:37.008433 17525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.130
I20250904 01:57:37.041646 17525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.112.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.130
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:37.042897 17525 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:37.044310 17525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:37.070695 17534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:37.192799 17521 heartbeater.cc:499] Master 127.16.112.190:43223 was elected leader, sending a full tablet report...
I20250904 01:57:37.302346 17538 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:37.302964 17538 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:37.305142 17538 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:37.306773 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:37.306880 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:37.309334 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:37.310639 17538 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:38.222450 17539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:38.223038 17539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:38.225248 17539 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:38.226143 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:38.226686 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:38.228264 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:38.229102 17539 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:38.245004 17540 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:38.245658 17540 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:38.248359 17540 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:38.248489 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:38.250397 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:38.251716 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:38.252671 17540 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:57:37.090615 17535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:38.286975 17525 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.209s user 0.417s sys 0.789s
W20250904 01:57:38.287286 17525 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.209s user 0.417s sys 0.789s
I20250904 01:57:38.289012 17525 server_base.cc:1047] running on GCE node
W20250904 01:57:38.290390 17542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:38.291391 17525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:38.293699 17525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:38.295099 17525 hybrid_clock.cc:648] HybridClock initialized: now 1756951058295064 us; error 32 us; skew 500 ppm
I20250904 01:57:38.295845 17525 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:38.302846 17525 webserver.cc:480] Webserver started at http://127.16.112.130:39979/ using document root <none> and password file <none>
I20250904 01:57:38.303696 17525 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:38.303887 17525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:38.304303 17525 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:38.308495 17525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/instance:
uuid: "c16f4aaad8b64d51a9df30717c732aa9"
format_stamp: "Formatted at 2025-09-04 01:57:38 on dist-test-slave-pl7p"
I20250904 01:57:38.309633 17525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance:
uuid: "c16f4aaad8b64d51a9df30717c732aa9"
format_stamp: "Formatted at 2025-09-04 01:57:38 on dist-test-slave-pl7p"
I20250904 01:57:38.317868 17525 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20250904 01:57:38.323295 17547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:38.324281 17525 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250904 01:57:38.324546 17525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
uuid: "c16f4aaad8b64d51a9df30717c732aa9"
format_stamp: "Formatted at 2025-09-04 01:57:38 on dist-test-slave-pl7p"
I20250904 01:57:38.324853 17525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:38.398888 17525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:38.400241 17525 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:38.400611 17525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:38.402904 17525 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:57:38.406447 17525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:57:38.406634 17525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:38.406852 17525 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:57:38.406989 17525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:38.535929 17525 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.130:40345
I20250904 01:57:38.536021 17659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.130:40345 every 8 connection(s)
I20250904 01:57:38.538317 17525 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
I20250904 01:57:38.545172 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17525
I20250904 01:57:38.545804 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance
I20250904 01:57:38.561764 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.131:0
--local_ip_for_outbound_sockets=127.16.112.131
--webserver_interface=127.16.112.131
--webserver_port=0
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--builtin_ntp_servers=127.16.112.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:57:38.571414 17661 heartbeater.cc:344] Connected to a master server at 127.16.112.189:39353
I20250904 01:57:38.571918 17661 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:38.573119 17660 heartbeater.cc:344] Connected to a master server at 127.16.112.188:43009
I20250904 01:57:38.573203 17661 heartbeater.cc:507] Master 127.16.112.189:39353 requested a full tablet report, sending...
I20250904 01:57:38.573432 17660 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:38.574218 17660 heartbeater.cc:507] Master 127.16.112.188:43009 requested a full tablet report, sending...
I20250904 01:57:38.576445 17184 ts_manager.cc:194] Registered new tserver with Master: c16f4aaad8b64d51a9df30717c732aa9 (127.16.112.130:40345)
I20250904 01:57:38.576517 17262 ts_manager.cc:194] Registered new tserver with Master: c16f4aaad8b64d51a9df30717c732aa9 (127.16.112.130:40345)
I20250904 01:57:38.583451 17662 heartbeater.cc:344] Connected to a master server at 127.16.112.190:43223
I20250904 01:57:38.583704 17662 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:38.584276 17662 heartbeater.cc:507] Master 127.16.112.190:43223 requested a full tablet report, sending...
I20250904 01:57:38.585965 17110 ts_manager.cc:194] Registered new tserver with Master: c16f4aaad8b64d51a9df30717c732aa9 (127.16.112.130:40345)
I20250904 01:57:38.587394 17110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.130:36405
I20250904 01:57:38.734557 17672 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:38.735013 17672 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:38.736812 17672 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:38.737412 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:38.738379 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:38.739925 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:38.740437 17672 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:57:38.814368 17673 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:38.814939 17673 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:38.816921 17673 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:38.817499 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:38.817883 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:38.818441 17271 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73ac0fbf752482195cd3a0207ae6eba in term 1.
I20250904 01:57:38.819470 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449, e73ac0fbf752482195cd3a0207ae6eba; no voters: d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:38.820057 17673 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:57:38.820353 17673 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:38.820662 17673 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:57:38.828578 17673 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:38.830241 17673 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 election: Requested vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:38.831120 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
I20250904 01:57:38.831229 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449"
I20250904 01:57:38.832441 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375
I20250904 01:57:38.832959 17673 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250904 01:57:38.868552 17670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:38.868973 17670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:38.869607 17670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:38.899685 17670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:38.900579 17670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.131
I20250904 01:57:38.933745 17670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.112.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:43223,127.16.112.189:39353,127.16.112.188:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.131
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:38.934966 17670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:38.936396 17670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:38.947412 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:39.090018 17662 heartbeater.cc:499] Master 127.16.112.190:43223 was elected leader, sending a full tablet report...
I20250904 01:57:39.226979 17193 raft_consensus.cc:1238] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Rejecting Update request from peer d8bba492ab9e479ba4a4ec6e37ece375 for earlier term 1. Current term is 2. Ops: []
I20250904 01:57:39.229199 17355 consensus_queue.cc:1046] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:57:39.231778 17685 raft_consensus.cc:3053] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 LEADER]: Stepping down as leader of term 1
I20250904 01:57:39.233217 17685 raft_consensus.cc:738] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8bba492ab9e479ba4a4ec6e37ece375, State: Running, Role: LEADER
I20250904 01:57:39.234439 17685 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.236471 17685 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:57:39.253324 17325 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:57:39.347584 17686 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:39.348555 17686 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.352123 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:39.352914 17120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616c90f4d4be4e11bb7b577e6528a449 in term 2.
I20250904 01:57:39.354468 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375; no voters:
I20250904 01:57:39.362296 17686 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:39.363885 17686 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:57:39.364874 17686 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8bba492ab9e479ba4a4ec6e37ece375)
I20250904 01:57:39.365731 17686 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:57:39.372584 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:39.373593 17193 raft_consensus.cc:2391] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 616c90f4d4be4e11bb7b577e6528a449 in current term 2: Already voted for candidate e73ac0fbf752482195cd3a0207ae6eba in this term.
I20250904 01:57:39.385871 17686 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.389001 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
I20250904 01:57:39.401273 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba"
I20250904 01:57:39.402148 17193 raft_consensus.cc:2391] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 616c90f4d4be4e11bb7b577e6528a449 in current term 2: Already voted for candidate e73ac0fbf752482195cd3a0207ae6eba in this term.
I20250904 01:57:39.404769 17120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616c90f4d4be4e11bb7b577e6528a449 in term 2.
I20250904 01:57:39.405283 17686 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 election: Requested vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:39.406664 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449, d8bba492ab9e479ba4a4ec6e37ece375; no voters: e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:39.408324 17686 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 01:57:39.409199 17686 raft_consensus.cc:695] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 LEADER]: Becoming Leader. State: Replica: 616c90f4d4be4e11bb7b577e6528a449, State: Running, Role: LEADER
I20250904 01:57:39.410356 17686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.452411 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616c90f4d4be4e11bb7b577e6528a449. Latest consensus state: current_term: 2 leader_uuid: "616c90f4d4be4e11bb7b577e6528a449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:39.453271 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:39.463536 17689 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:57:39.474514 17689 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:57:39.479630 17689 catalog_manager.cc:1261] Loaded cluster ID: ff74342adf9a48c0ac0b7771a1ce3900
I20250904 01:57:39.480026 17689 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:57:39.485762 17689 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:57:39.490842 17689 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Loaded TSK: 0
I20250904 01:57:39.492489 17689 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:57:39.497505 17689 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:57:39.506592 17690 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:57:39.507304 17690 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.509702 17690 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:39.512787 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:39.513445 17120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73ac0fbf752482195cd3a0207ae6eba in term 2.
I20250904 01:57:39.516345 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449" is_pre_election: true
I20250904 01:57:39.522368 17689 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:57:39.519981 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba; no voters:
I20250904 01:57:39.524391 17691 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 01:57:39.526221 17691 raft_consensus.cc:491] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:57:39.527148 17691 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:57:39.536983 17691 raft_consensus.cc:513] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.546231 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375"
I20250904 01:57:39.546864 17691 leader_election.cc:290] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 3 election: Requested vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), 616c90f4d4be4e11bb7b577e6528a449 (127.16.112.188:43009)
I20250904 01:57:39.546901 17120 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:57:39.550498 17271 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e73ac0fbf752482195cd3a0207ae6eba" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "616c90f4d4be4e11bb7b577e6528a449"
I20250904 01:57:39.561281 17120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e73ac0fbf752482195cd3a0207ae6eba in term 3.
I20250904 01:57:39.562357 17169 leader_election.cc:304] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba; no voters: 616c90f4d4be4e11bb7b577e6528a449
I20250904 01:57:39.563266 17691 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 3 FOLLOWER]: Leader election won for term 3
I20250904 01:57:39.563721 17691 raft_consensus.cc:695] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 3 LEADER]: Becoming Leader. State: Replica: e73ac0fbf752482195cd3a0207ae6eba, State: Running, Role: LEADER
I20250904 01:57:39.564561 17691 consensus_queue.cc:237] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.571156 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73ac0fbf752482195cd3a0207ae6eba. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:39.572124 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:39.578155 17693 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:57:39.592422 17693 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:57:39.597249 17693 catalog_manager.cc:1261] Loaded cluster ID: ff74342adf9a48c0ac0b7771a1ce3900
I20250904 01:57:39.601605 17693 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:57:39.606135 17693 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:57:39.614032 17693 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba: Loaded TSK: 0
I20250904 01:57:39.615581 17693 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:57:39.620465 17693 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:57:39.625087 17693 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:57:39.876807 17193 raft_consensus.cc:1238] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [term 3 LEADER]: Rejecting Update request from peer 616c90f4d4be4e11bb7b577e6528a449 for earlier term 2. Current term is 3. Ops: []
I20250904 01:57:39.878913 17687 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:57:39.879896 17687 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [LEADER]: Peer e73ac0fbf752482195cd3a0207ae6eba log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250904 01:57:39.880434 17687 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 LEADER]: Stepping down as leader of term 2
I20250904 01:57:39.880779 17687 raft_consensus.cc:738] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 616c90f4d4be4e11bb7b577e6528a449, State: Running, Role: LEADER
I20250904 01:57:39.882087 17687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:39.884413 17687 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:57:39.945864 17271 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 3 FOLLOWER]: Refusing update from remote peer e73ac0fbf752482195cd3a0207ae6eba: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250904 01:57:39.946275 17271 pending_rounds.cc:85] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Aborting all ops after (but not including) 4
I20250904 01:57:39.946594 17271 pending_rounds.cc:107] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250904 01:57:39.946877 17271 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250904 01:57:39.949301 17690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:57:39.970840 17120 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [term 3 FOLLOWER]: Refusing update from remote peer e73ac0fbf752482195cd3a0207ae6eba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 01:57:39.970645 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:39.971300 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:39.972699 17690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250904 01:57:39.980998 17686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:39.981801 17686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:39.983078 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73ac0fbf752482195cd3a0207ae6eba. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:39.983943 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:40.027567 17697 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73ac0fbf752482195cd3a0207ae6eba. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:40.028498 17697 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:40.031868 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:40.032744 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bba492ab9e479ba4a4ec6e37ece375 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:57:40.093245 17660 heartbeater.cc:499] Master 127.16.112.188:43009 was elected leader, sending a full tablet report...
I20250904 01:57:40.187333 17517 heartbeater.cc:499] Master 127.16.112.188:43009 was elected leader, sending a full tablet report...
I20250904 01:57:40.192653 17518 heartbeater.cc:499] Master 127.16.112.189:39353 was elected leader, sending a full tablet report...
I20250904 01:57:40.462586 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e73ac0fbf752482195cd3a0207ae6eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } } }
I20250904 01:57:40.463231 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73ac0fbf752482195cd3a0207ae6eba [sys.catalog]: This master's current role is: LEADER
I20250904 01:57:40.599421 17661 heartbeater.cc:499] Master 127.16.112.189:39353 was elected leader, sending a full tablet report...
W20250904 01:57:40.351094 17678 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 17670
W20250904 01:57:40.661257 17670 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.714s user 0.622s sys 1.052s
W20250904 01:57:40.661816 17670 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.714s user 0.622s sys 1.052s
W20250904 01:57:38.948505 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:40.662925 17681 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1714 milliseconds
I20250904 01:57:40.663399 17670 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 01:57:40.663483 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:40.666677 17670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:40.669109 17670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:40.670533 17670 hybrid_clock.cc:648] HybridClock initialized: now 1756951060670488 us; error 56 us; skew 500 ppm
I20250904 01:57:40.671331 17670 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:40.676656 17670 webserver.cc:480] Webserver started at http://127.16.112.131:33859/ using document root <none> and password file <none>
I20250904 01:57:40.677465 17670 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:40.677711 17670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:40.678071 17670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:40.682163 17670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/instance:
uuid: "39984684ed35439696da34e2927a871d"
format_stamp: "Formatted at 2025-09-04 01:57:40 on dist-test-slave-pl7p"
I20250904 01:57:40.683164 17670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance:
uuid: "39984684ed35439696da34e2927a871d"
format_stamp: "Formatted at 2025-09-04 01:57:40 on dist-test-slave-pl7p"
I20250904 01:57:40.689524 17670 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250904 01:57:40.694919 17706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:40.695987 17670 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250904 01:57:40.696269 17670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
uuid: "39984684ed35439696da34e2927a871d"
format_stamp: "Formatted at 2025-09-04 01:57:40 on dist-test-slave-pl7p"
I20250904 01:57:40.696569 17670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:40.772111 17670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:40.773458 17670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:40.773959 17670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:40.776261 17670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:57:40.780052 17670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:57:40.780280 17670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:40.780515 17670 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:57:40.780669 17670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:40.906054 17670 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.131:37027
I20250904 01:57:40.906142 17818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.131:37027 every 8 connection(s)
I20250904 01:57:40.908501 17670 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
I20250904 01:57:40.913352 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 17670
I20250904 01:57:40.913868 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance
I20250904 01:57:40.942662 17820 heartbeater.cc:344] Connected to a master server at 127.16.112.189:39353
I20250904 01:57:40.943094 17820 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:40.943346 17819 heartbeater.cc:344] Connected to a master server at 127.16.112.188:43009
I20250904 01:57:40.943624 17819 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:40.944607 17820 heartbeater.cc:507] Master 127.16.112.189:39353 requested a full tablet report, sending...
I20250904 01:57:40.944656 17819 heartbeater.cc:507] Master 127.16.112.188:43009 requested a full tablet report, sending...
I20250904 01:57:40.942159 17829 raft_consensus.cc:491] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e73ac0fbf752482195cd3a0207ae6eba)
I20250904 01:57:40.946264 17829 raft_consensus.cc:513] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 43223 } } peers { permanent_uuid: "e73ac0fbf752482195cd3a0207ae6eba" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 39353 } } peers { permanent_uuid: "616c90f4d4be4e11bb7b577e6528a449" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 43009 } }
I20250904 01:57:40.947431 17184 ts_manager.cc:194] Registered new tserver with Master: 39984684ed35439696da34e2927a871d (127.16.112.131:37027)
I20250904 01:57:40.948047 17262 ts_manager.cc:194] Registered new tserver with Master: 39984684ed35439696da34e2927a871d (127.16.112.131:37027)
I20250904 01:57:40.950186 17184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.131:55247
I20250904 01:57:40.950986 17120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8bba492ab9e479ba4a4ec6e37ece375" is_pre_election: true
I20250904 01:57:40.951120 17821 heartbeater.cc:344] Connected to a master server at 127.16.112.190:43223
I20250904 01:57:40.951438 17821 heartbeater.cc:461] Registering TS with master...
I20250904 01:57:40.952268 17821 heartbeater.cc:507] Master 127.16.112.190:43223 requested a full tablet report, sending...
I20250904 01:57:40.953425 17829 leader_election.cc:290] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8bba492ab9e479ba4a4ec6e37ece375 (127.16.112.190:43223), e73ac0fbf752482195cd3a0207ae6eba (127.16.112.189:39353)
I20250904 01:57:40.953924 17193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "616c90f4d4be4e11bb7b577e6528a449" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e73ac0fbf752482195cd3a0207ae6eba" is_pre_election: true
I20250904 01:57:40.954970 17110 ts_manager.cc:194] Registered new tserver with Master: 39984684ed35439696da34e2927a871d (127.16.112.131:37027)
I20250904 01:57:40.955119 17247 leader_election.cc:304] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616c90f4d4be4e11bb7b577e6528a449; no voters: d8bba492ab9e479ba4a4ec6e37ece375, e73ac0fbf752482195cd3a0207ae6eba
I20250904 01:57:40.955857 17829 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 616c90f4d4be4e11bb7b577e6528a449 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:57:40.968863 16834 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master_failover-itest.cc:178: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250904 01:57:41.005728 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17367
I20250904 01:57:41.026434 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17525
I20250904 01:57:41.051609 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17670
I20250904 01:57:41.075146 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17077
I20250904 01:57:41.112854 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17146
I20250904 01:57:41.139302 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 17220
W20250904 01:57:42.170027 16834 thread.cc:527] Waited for 1000ms trying to join with rpc reactor (tid 16840)
Shutting down hive metastore.
2025-09-04T01:57:42Z chronyd exiting
[ SKIPPED ] HmsConfigurations/MasterFailoverTest.TestCreateTableSync/1 (29068 ms)
[ RUN ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1
2025-09-04T01:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T01:57:42Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
2025-09-04 01:57:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250904 01:57:56.105639 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:46159
--webserver_interface=127.16.112.190
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--hive_metastore_uris=thrift://127.0.0.1:35799
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
W20250904 01:57:56.388144 18080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:56.388711 18080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:56.389146 18080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:56.418650 18080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:57:56.418962 18080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:56.419205 18080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:57:56.419428 18080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:57:56.454212 18080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35799
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.112.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:56.455523 18080 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:56.457110 18080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:56.467006 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:57.871093 18085 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 18080
W20250904 01:57:56.467557 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:58.233070 18080 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.766s user 0.560s sys 1.206s
W20250904 01:57:58.234416 18080 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.767s user 0.560s sys 1.206s
W20250904 01:57:58.234944 18088 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1766 milliseconds
I20250904 01:57:58.235291 18080 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 01:57:58.235378 18089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:58.238842 18080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:58.241183 18080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:58.242502 18080 hybrid_clock.cc:648] HybridClock initialized: now 1756951078242472 us; error 33 us; skew 500 ppm
I20250904 01:57:58.243283 18080 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:58.249007 18080 webserver.cc:480] Webserver started at http://127.16.112.190:42327/ using document root <none> and password file <none>
I20250904 01:57:58.249927 18080 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:58.250136 18080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:58.250558 18080 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:58.254801 18080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/instance:
uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
format_stamp: "Formatted at 2025-09-04 01:57:58 on dist-test-slave-pl7p"
I20250904 01:57:58.255829 18080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance:
uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
format_stamp: "Formatted at 2025-09-04 01:57:58 on dist-test-slave-pl7p"
I20250904 01:57:58.262454 18080 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20250904 01:57:58.267524 18096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:58.268409 18080 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250904 01:57:58.268713 18080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
format_stamp: "Formatted at 2025-09-04 01:57:58 on dist-test-slave-pl7p"
I20250904 01:57:58.269023 18080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:57:58.319672 18080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:57:58.321095 18080 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:57:58.321516 18080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:57:58.388116 18080 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.190:46159
I20250904 01:57:58.388180 18147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.190:46159 every 8 connection(s)
I20250904 01:57:58.390688 18080 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
I20250904 01:57:58.391595 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18080
I20250904 01:57:58.391990 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance
I20250904 01:57:58.394078 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:38643
--webserver_interface=127.16.112.189
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--hive_metastore_uris=thrift://127.0.0.1:35799
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:57:58.395681 18148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:57:58.405236 18148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:57:58.435936 18148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:57:58.438155 18098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
W20250904 01:57:58.442188 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
I20250904 01:57:58.491613 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 1
W20250904 01:57:58.495764 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
I20250904 01:57:58.564131 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 2
W20250904 01:57:58.568253 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
I20250904 01:57:58.659607 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 3
W20250904 01:57:58.663544 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
W20250904 01:57:58.689422 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:57:58.690214 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:57:58.690680 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:57:58.720641 18149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:57:58.720963 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:57:58.721211 18149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:57:58.721451 18149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:57:58.754381 18149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:35799
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.112.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:57:58.755671 18149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:57:58.757236 18149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:57:58.767266 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:57:58.806926 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 4
W20250904 01:57:58.813091 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
I20250904 01:57:59.112641 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 5
W20250904 01:57:59.118741 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
I20250904 01:57:59.666347 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 6
W20250904 01:57:59.677156 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38643: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38643: connect: Connection refused (error 111)
W20250904 01:57:58.771375 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:58.768110 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:57:59.953369 18160 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1181 milliseconds
I20250904 01:57:59.953471 18149 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:57:59.954655 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:57:59.957500 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:57:59.958905 18149 hybrid_clock.cc:648] HybridClock initialized: now 1756951079958872 us; error 53 us; skew 500 ppm
I20250904 01:57:59.959656 18149 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:57:59.966048 18149 webserver.cc:480] Webserver started at http://127.16.112.189:45103/ using document root <none> and password file <none>
I20250904 01:57:59.966943 18149 fs_manager.cc:362] Metadata directory not provided
I20250904 01:57:59.967135 18149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:57:59.967548 18149 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:57:59.971860 18149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/instance:
uuid: "cac61d83403b4e51875bc24a3a91a3eb"
format_stamp: "Formatted at 2025-09-04 01:57:59 on dist-test-slave-pl7p"
I20250904 01:57:59.972940 18149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance:
uuid: "cac61d83403b4e51875bc24a3a91a3eb"
format_stamp: "Formatted at 2025-09-04 01:57:59 on dist-test-slave-pl7p"
I20250904 01:57:59.979676 18149 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250904 01:57:59.984803 18170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:57:59.985795 18149 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:57:59.986074 18149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
uuid: "cac61d83403b4e51875bc24a3a91a3eb"
format_stamp: "Formatted at 2025-09-04 01:57:59 on dist-test-slave-pl7p"
I20250904 01:57:59.986364 18149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:00.027752 18149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:00.028988 18149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:00.029366 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:00.095322 18149 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.189:38643
I20250904 01:58:00.095446 18221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.189:38643 every 8 connection(s)
I20250904 01:58:00.097827 18149 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
I20250904 01:58:00.100840 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18149
I20250904 01:58:00.101197 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance
I20250904 01:58:00.102638 18222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:00.103397 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:35271
--webserver_interface=127.16.112.188
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--hive_metastore_uris=thrift://127.0.0.1:35799
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:58:00.112231 18222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:00.131582 18222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:00.144959 18222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:00.147001 18172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
W20250904 01:58:00.150764 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.200305 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 1
W20250904 01:58:00.204048 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.272562 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 2
W20250904 01:58:00.276355 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.367830 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 3
W20250904 01:58:00.372644 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
W20250904 01:58:00.418856 18223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:00.419422 18223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:00.419876 18223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:00.450917 18223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:58:00.451231 18223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:00.451473 18223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:58:00.451715 18223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:58:00.486968 18223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35799
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.112.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:00.488260 18223 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:00.489816 18223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:00.499507 18233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:00.516070 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 4
W20250904 01:58:00.523191 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.737692 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } attempt: 7
I20250904 01:58:00.784827 18148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:00.810379 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.822698 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 5
W20250904 01:58:00.830253 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.868990 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 1
W20250904 01:58:00.876930 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:00.958498 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 2
W20250904 01:58:00.966044 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:01.051862 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 3
W20250904 01:58:01.059401 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:01.200122 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 4
W20250904 01:58:01.207998 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:01.377669 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 6
W20250904 01:58:01.405277 18222 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:01.491573 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 5
W20250904 01:58:01.498517 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
I20250904 01:58:02.051025 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 6
W20250904 01:58:02.083209 18148 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:35271: Network error: Client connection negotiation failed: client connection to 127.16.112.188:35271: connect: Connection refused (error 111)
W20250904 01:58:01.904721 18232 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 18223
W20250904 01:58:02.090245 18223 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.590s user 0.563s sys 1.012s
W20250904 01:58:00.501284 18234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:02.090711 18223 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.590s user 0.564s sys 1.013s
W20250904 01:58:02.092459 18236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:02.098031 18235 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1591 milliseconds
I20250904 01:58:02.098107 18223 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:02.099562 18223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:02.102734 18223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:02.104267 18223 hybrid_clock.cc:648] HybridClock initialized: now 1756951082104231 us; error 27 us; skew 500 ppm
I20250904 01:58:02.105319 18223 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:02.115747 18223 webserver.cc:480] Webserver started at http://127.16.112.188:40255/ using document root <none> and password file <none>
I20250904 01:58:02.116607 18223 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:02.116845 18223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:02.117286 18223 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:02.121613 18223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/instance:
uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6"
format_stamp: "Formatted at 2025-09-04 01:58:02 on dist-test-slave-pl7p"
I20250904 01:58:02.122656 18223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance:
uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6"
format_stamp: "Formatted at 2025-09-04 01:58:02 on dist-test-slave-pl7p"
I20250904 01:58:02.129341 18223 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250904 01:58:02.134610 18251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:02.135526 18223 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250904 01:58:02.135818 18223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6"
format_stamp: "Formatted at 2025-09-04 01:58:02 on dist-test-slave-pl7p"
I20250904 01:58:02.136137 18223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:02.197443 18223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:02.198856 18223 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:02.199268 18223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:02.266182 18223 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.188:35271
I20250904 01:58:02.266244 18302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.188:35271 every 8 connection(s)
I20250904 01:58:02.268738 18223 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
I20250904 01:58:02.273324 18303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:02.277697 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18223
I20250904 01:58:02.278066 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance
I20250904 01:58:02.285629 18303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:02.311852 18303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:02.320710 18303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:02.335165 18303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: Bootstrap starting.
I20250904 01:58:02.340600 18303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:02.342722 18303 log.cc:826] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:02.346660 18303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: No bootstrap required, opened a new log
I20250904 01:58:02.364766 18303 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.365381 18303 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:02.365592 18303 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4904879f942d4b9f8a9fb8f5de6cfbe6, State: Initialized, Role: FOLLOWER
I20250904 01:58:02.366202 18303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.368335 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.368986 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:02.370502 18303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: configured and running, proceeding with master startup.
W20250904 01:58:02.405750 18325 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:58:02.406170 18325 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:58:02.402809 18223 master_runner.cc:186] Error getting master registration for 127.16.112.190:46159: OK, instance_id { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" instance_seqno: 1756951078370785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:58:02.430034 18223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:58:02.453977 18257 tablet.cc:2378] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:58:02.465763 18222 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 7
I20250904 01:58:02.487532 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: Bootstrap starting.
I20250904 01:58:02.495743 18222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:02.498101 18222 log.cc:826] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:02.503379 18222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: No bootstrap required, opened a new log
I20250904 01:58:02.535871 18222 raft_consensus.cc:357] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.536770 18222 raft_consensus.cc:383] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:02.537149 18222 raft_consensus.cc:738] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac61d83403b4e51875bc24a3a91a3eb, State: Initialized, Role: FOLLOWER
I20250904 01:58:02.538113 18222 consensus_queue.cc:260] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.540510 18330 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.541107 18330 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:02.542732 18222 sys_catalog.cc:564] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: configured and running, proceeding with master startup.
W20250904 01:58:02.589411 18345 catalog_manager.cc:1560] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:58:02.589855 18345 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:58:02.572747 18149 master_runner.cc:186] Error getting master registration for 127.16.112.190:46159: OK, instance_id { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" instance_seqno: 1756951078370785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:58:02.609323 18149 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250904 01:58:02.625442 18330 raft_consensus.cc:491] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:02.626065 18330 raft_consensus.cc:513] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.630215 18330 leader_election.cc:290] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:02.630666 18278 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:02.631518 18278 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 0.
I20250904 01:58:02.632735 18172 leader_election.cc:304] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb; no voters:
I20250904 01:58:02.633837 18330 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:58:02.634155 18330 raft_consensus.cc:491] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:02.634465 18330 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:02.641088 18330 raft_consensus.cc:513] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.643848 18278 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6"
I20250904 01:58:02.644366 18278 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:02.651634 18278 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 1.
I20250904 01:58:02.652601 18172 leader_election.cc:304] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb; no voters:
I20250904 01:58:02.652315 18330 leader_election.cc:290] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 1 election: Requested vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:02.660037 18330 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:58:02.661096 18330 raft_consensus.cc:695] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 LEADER]: Becoming Leader. State: Replica: cac61d83403b4e51875bc24a3a91a3eb, State: Running, Role: LEADER
I20250904 01:58:02.662611 18330 consensus_queue.cc:237] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:02.672971 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.673866 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:02.677218 18349 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:02.684492 18349 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:02.708787 18278 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Refusing update from remote peer cac61d83403b4e51875bc24a3a91a3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:58:02.711967 18347 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:58:02.763592 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.767608 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:02.774891 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.777191 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:02.786562 18349 catalog_manager.cc:1349] Generated new cluster ID: 360389147c5748c9925cf97041f6ced8
I20250904 01:58:02.786865 18349 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:02.794910 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:02.795574 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:02.836858 18349 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 01:58:02.838824 18349 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:02.873411 18349 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: Generated new TSK 0
I20250904 01:58:02.874449 18349 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:02.878558 18349 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:02.882669 18349 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:58:02.907684 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.129:0
--local_ip_for_outbound_sockets=127.16.112.129
--webserver_interface=127.16.112.129
--webserver_port=0
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:03.116691 18148 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } attempt: 7
I20250904 01:58:03.181211 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: Bootstrap starting.
I20250904 01:58:03.200485 18148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:03.207356 18148 log.cc:826] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:03.220937 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: No bootstrap required, opened a new log
I20250904 01:58:03.412510 18325 catalog_manager.cc:1261] Loaded cluster ID: 360389147c5748c9925cf97041f6ced8
I20250904 01:58:03.412869 18325 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: loading cluster ID for follower catalog manager: success
I20250904 01:58:03.419950 18148 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:03.420748 18148 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:03.421046 18148 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5748fb924edd4e1b8a52330cbe5d3d5b, State: Initialized, Role: FOLLOWER
I20250904 01:58:03.421950 18148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:03.419669 18361 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:03.425199 18325 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: acquiring CA information for follower catalog manager: success
I20250904 01:58:03.434567 18325 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:58:03.437551 18365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:03.428196 18361 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:03.444543 18365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:03.452172 18148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:58:03.452378 18122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
I20250904 01:58:03.453346 18122 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:03.458484 18123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
W20250904 01:58:03.459709 18123 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35422 (request call id 2) took 828 ms (client timeout 670 ms). Trace:
W20250904 01:58:03.460127 18123 rpcz_store.cc:269] 0904 01:58:02.631487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0904 01:58:02.631596 (+ 109us) service_pool.cc:225] Handling call
0904 01:58:03.459607 (+828011us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250904 01:58:03.461408 18119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:03.462879 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
I20250904 01:58:03.470433 18122 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 1.
I20250904 01:58:03.475673 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:03.473290 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:03.478904 18361 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
W20250904 01:58:03.485921 18122 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35422 (request call id 3) took 842 ms (client timeout 546 ms). Trace:
W20250904 01:58:03.486522 18122 rpcz_store.cc:269] 0904 01:58:02.643543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0904 01:58:02.649687 (+ 6144us) service_pool.cc:225] Handling call
0904 01:58:03.485731 (+836044us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250904 01:58:03.488116 18121 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Refusing update from remote peer cac61d83403b4e51875bc24a3a91a3eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:58:03.494463 18255 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:03.495599 18361 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:03.497032 18362 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 01:58:03.543315 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:03.543980 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:03.544924 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250904 01:58:03.547528 18365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:03.547600 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:03.548233 18365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:03.548571 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:03.593088 18387 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250904 01:58:03.600870 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:03.602476 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.129
I20250904 01:58:03.673023 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.112.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.129
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:03.674396 18356 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:03.675940 18356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:03.689280 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:03.943979 18362 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:03.944905 18362 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:04.395887 18397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:04.396586 18397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:04.399296 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:04.414008 18397 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:04.415009 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:04.417011 18253 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:04.417773 18397 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:04.447853 18398 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:04.448807 18398 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:04.452180 18398 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:04.452677 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:04.455324 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:04.456845 18098 leader_election.cc:304] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:04.459425 18398 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:04.619668 18387 catalog_manager.cc:1261] Loaded cluster ID: 360389147c5748c9925cf97041f6ced8
I20250904 01:58:04.626091 18387 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: loading cluster ID for follower catalog manager: success
I20250904 01:58:04.661136 18387 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: acquiring CA information for follower catalog manager: success
I20250904 01:58:04.687418 18387 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 01:58:03.691458 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:05.091558 18388 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 18356
W20250904 01:58:05.381139 18388 kernel_stack_watchdog.cc:198] Thread 18356 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 01:58:05.381671 18356 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.693s user 0.401s sys 0.700s
W20250904 01:58:05.381997 18356 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.693s user 0.401s sys 0.700s
I20250904 01:58:05.397310 18356 server_base.cc:1047] running on GCE node
I20250904 01:58:05.398015 18400 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:05.398520 18400 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:05.400693 18400 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:05.401043 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
W20250904 01:58:05.402459 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:05.402297 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:05.403877 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:05.404383 18253 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:05.405398 18400 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:05.412653 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:05.420409 18356 hybrid_clock.cc:648] HybridClock initialized: now 1756951085419118 us; error 1300 us; skew 500 ppm
I20250904 01:58:05.421403 18356 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:05.440526 18356 webserver.cc:480] Webserver started at http://127.16.112.129:46693/ using document root <none> and password file <none>
I20250904 01:58:05.441763 18356 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:05.442040 18356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:05.442580 18356 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:05.448789 18356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/instance:
uuid: "e260898a559747088b33446ac00ec800"
format_stamp: "Formatted at 2025-09-04 01:58:05 on dist-test-slave-pl7p"
I20250904 01:58:05.450196 18356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance:
uuid: "e260898a559747088b33446ac00ec800"
format_stamp: "Formatted at 2025-09-04 01:58:05 on dist-test-slave-pl7p"
I20250904 01:58:05.459563 18356 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250904 01:58:05.466890 18407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:05.468127 18356 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250904 01:58:05.468477 18356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
uuid: "e260898a559747088b33446ac00ec800"
format_stamp: "Formatted at 2025-09-04 01:58:05 on dist-test-slave-pl7p"
I20250904 01:58:05.468919 18356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:05.540354 18356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:05.542225 18356 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:05.542749 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:05.546854 18356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:05.552361 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:05.552640 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:05.552934 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:05.553145 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:05.708741 18356 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.129:42343
I20250904 01:58:05.708822 18519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.129:42343 every 8 connection(s)
I20250904 01:58:05.712308 18356 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
I20250904 01:58:05.714926 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18356
I20250904 01:58:05.715618 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance
I20250904 01:58:05.728096 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.130:0
--local_ip_for_outbound_sockets=127.16.112.130
--webserver_interface=127.16.112.130
--webserver_port=0
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:05.775565 18520 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38643
I20250904 01:58:05.776057 18520 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:05.777318 18520 heartbeater.cc:507] Master 127.16.112.189:38643 requested a full tablet report, sending...
I20250904 01:58:05.780597 18187 ts_manager.cc:194] Registered new tserver with Master: e260898a559747088b33446ac00ec800 (127.16.112.129:42343)
I20250904 01:58:05.783247 18187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.129:40371
I20250904 01:58:05.789623 18521 heartbeater.cc:344] Connected to a master server at 127.16.112.188:35271
I20250904 01:58:05.789971 18521 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:05.790593 18521 heartbeater.cc:507] Master 127.16.112.188:35271 requested a full tablet report, sending...
I20250904 01:58:05.793354 18268 ts_manager.cc:194] Registered new tserver with Master: e260898a559747088b33446ac00ec800 (127.16.112.129:42343)
I20250904 01:58:05.796636 18525 heartbeater.cc:344] Connected to a master server at 127.16.112.190:46159
I20250904 01:58:05.796979 18525 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:05.797616 18525 heartbeater.cc:507] Master 127.16.112.190:46159 requested a full tablet report, sending...
I20250904 01:58:05.800379 18113 ts_manager.cc:194] Registered new tserver with Master: e260898a559747088b33446ac00ec800 (127.16.112.129:42343)
I20250904 01:58:05.942265 18531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:05.942757 18531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:05.944734 18531 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:05.945825 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:05.946942 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:05.948374 18255 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:05.949095 18531 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:06.023236 18532 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:06.023850 18532 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:06.025938 18532 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:06.027184 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:06.027284 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:06.028928 18098 leader_election.cc:304] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:06.029650 18532 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:58:06.077185 18524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:06.077715 18524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:06.078392 18524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:06.109501 18524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:06.110471 18524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.130
I20250904 01:58:06.145902 18524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.112.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.130
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:06.147190 18524 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:06.148667 18524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:06.161592 18538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:06.287269 18520 heartbeater.cc:499] Master 127.16.112.189:38643 was elected leader, sending a full tablet report...
I20250904 01:58:07.023804 18545 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:07.024672 18545 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:07.028488 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:07.032650 18545 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:07.032819 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:07.034724 18253 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:07.036083 18545 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:07.417333 18546 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:07.418246 18546 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:07.422039 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:07.429991 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:07.435649 18098 leader_election.cc:304] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:07.442613 18546 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:07.444343 18546 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:58:06.165807 18541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:06.163130 18539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:07.689328 18540 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1522 milliseconds
I20250904 01:58:07.689508 18524 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:07.691030 18524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:07.693722 18524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:07.695142 18524 hybrid_clock.cc:648] HybridClock initialized: now 1756951087695095 us; error 53 us; skew 500 ppm
I20250904 01:58:07.696156 18524 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:07.703908 18524 webserver.cc:480] Webserver started at http://127.16.112.130:35175/ using document root <none> and password file <none>
I20250904 01:58:07.704820 18524 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:07.705044 18524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:07.705657 18524 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:07.712471 18524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/instance:
uuid: "8cfdaaf0e4a84ad1afe2ec955bb94c2c"
format_stamp: "Formatted at 2025-09-04 01:58:07 on dist-test-slave-pl7p"
I20250904 01:58:07.714007 18524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance:
uuid: "8cfdaaf0e4a84ad1afe2ec955bb94c2c"
format_stamp: "Formatted at 2025-09-04 01:58:07 on dist-test-slave-pl7p"
I20250904 01:58:07.723779 18524 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s
I20250904 01:58:07.731143 18552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:07.732306 18524 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250904 01:58:07.732678 18524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
uuid: "8cfdaaf0e4a84ad1afe2ec955bb94c2c"
format_stamp: "Formatted at 2025-09-04 01:58:07 on dist-test-slave-pl7p"
I20250904 01:58:07.733191 18524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:07.820468 18524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:07.822348 18524 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:07.822849 18524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:07.825898 18524 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:07.831040 18524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:07.831295 18524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:07.831599 18524 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:07.831813 18524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:07.980348 18524 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.130:39853
I20250904 01:58:07.980417 18664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.130:39853 every 8 connection(s)
I20250904 01:58:07.983732 18524 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
I20250904 01:58:07.991794 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18524
I20250904 01:58:07.992290 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance
I20250904 01:58:08.019508 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.131:0
--local_ip_for_outbound_sockets=127.16.112.131
--webserver_interface=127.16.112.131
--webserver_port=0
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--builtin_ntp_servers=127.16.112.148:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:08.026191 18675 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:08.026692 18675 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.028971 18675 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:08.030454 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:08.031590 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:08.033610 18098 leader_election.cc:304] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:08.035578 18675 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:08.037391 18665 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38643
I20250904 01:58:08.037932 18665 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:08.039026 18667 heartbeater.cc:344] Connected to a master server at 127.16.112.190:46159
I20250904 01:58:08.039238 18665 heartbeater.cc:507] Master 127.16.112.189:38643 requested a full tablet report, sending...
I20250904 01:58:08.039377 18667 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:08.040234 18667 heartbeater.cc:507] Master 127.16.112.190:46159 requested a full tablet report, sending...
I20250904 01:58:08.042646 18187 ts_manager.cc:194] Registered new tserver with Master: 8cfdaaf0e4a84ad1afe2ec955bb94c2c (127.16.112.130:39853)
I20250904 01:58:08.043062 18113 ts_manager.cc:194] Registered new tserver with Master: 8cfdaaf0e4a84ad1afe2ec955bb94c2c (127.16.112.130:39853)
I20250904 01:58:08.044284 18187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.130:42127
I20250904 01:58:08.045003 18666 heartbeater.cc:344] Connected to a master server at 127.16.112.188:35271
I20250904 01:58:08.045323 18666 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:08.046073 18666 heartbeater.cc:507] Master 127.16.112.188:35271 requested a full tablet report, sending...
I20250904 01:58:08.047955 18268 ts_manager.cc:194] Registered new tserver with Master: 8cfdaaf0e4a84ad1afe2ec955bb94c2c (127.16.112.130:39853)
I20250904 01:58:08.082409 18677 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:08.082906 18677 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.084470 18677 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:08.085230 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:08.085601 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:08.085847 18120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4904879f942d4b9f8a9fb8f5de6cfbe6 in term 1.
I20250904 01:58:08.087029 18255 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:08.087618 18677 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:58:08.087958 18677 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:08.088279 18677 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:58:08.094729 18677 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.096419 18677 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 election: Requested vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:08.097321 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
I20250904 01:58:08.098347 18277 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 2 FOLLOWER]: Rejecting Update request from peer cac61d83403b4e51875bc24a3a91a3eb for earlier term 1. Current term is 2. Ops: []
I20250904 01:58:08.099241 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb"
I20250904 01:58:08.100164 18678 consensus_queue.cc:1046] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250904 01:58:08.100600 18253 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:08.101711 18677 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250904 01:58:08.104950 18679 raft_consensus.cc:3053] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 LEADER]: Stepping down as leader of term 1
I20250904 01:58:08.105257 18679 raft_consensus.cc:738] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cac61d83403b4e51875bc24a3a91a3eb, State: Running, Role: LEADER
I20250904 01:58:08.106009 18679 consensus_queue.cc:260] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.107163 18679 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 1 FOLLOWER]: Advancing to term 2
W20250904 01:58:08.341288 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:08.341843 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:08.342424 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:08.374009 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:08.374857 18676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.131
I20250904 01:58:08.416927 18676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.112.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:46159,127.16.112.189:38643,127.16.112.188:35271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.131
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:08.418609 18676 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:08.420687 18676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:08.434989 18685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:08.549228 18665 heartbeater.cc:499] Master 127.16.112.189:38643 was elected leader, sending a full tablet report...
I20250904 01:58:08.606379 18345 catalog_manager.cc:1604] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:58:08.614382 18679 raft_consensus.cc:491] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:08.615068 18679 raft_consensus.cc:513] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.619398 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:08.620277 18120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 1.
I20250904 01:58:08.622188 18174 leader_election.cc:304] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb; no voters:
I20250904 01:58:08.624043 18679 leader_election.cc:290] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:08.624714 18679 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 01:58:08.625118 18679 raft_consensus.cc:491] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:08.625264 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:08.625541 18679 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:58:08.626029 18277 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 2.
I20250904 01:58:08.648725 18691 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:08.649545 18691 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.651345 18679 raft_consensus.cc:513] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.655961 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:08.657814 18197 raft_consensus.cc:2366] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5748fb924edd4e1b8a52330cbe5d3d5b for earlier term 2. Current term is 3.
I20250904 01:58:08.659888 18098 leader_election.cc:400] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Vote denied by peer cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5748fb924edd4e1b8a52330cbe5d3d5b for earlier term 2. Current term is 3.
I20250904 01:58:08.660300 18098 leader_election.cc:403] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250904 01:58:08.665789 18691 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:08.669505 18691 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 1 FOLLOWER]: Advancing to term 3
I20250904 01:58:08.681484 18679 leader_election.cc:290] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 3 election: Requested vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:08.682825 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6"
I20250904 01:58:08.683581 18277 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:58:08.693193 18277 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 3.
I20250904 01:58:08.696992 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cac61d83403b4e51875bc24a3a91a3eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b"
I20250904 01:58:08.699863 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:08.700977 18277 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5748fb924edd4e1b8a52330cbe5d3d5b for earlier term 2. Current term is 3.
I20250904 01:58:08.703305 18098 leader_election.cc:400] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5748fb924edd4e1b8a52330cbe5d3d5b for earlier term 2. Current term is 3.
I20250904 01:58:08.708432 18691 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5748fb924edd4e1b8a52330cbe5d3d5b for earlier term 2. Current term is 3.
I20250904 01:58:08.718758 18172 leader_election.cc:304] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb; no voters:
I20250904 01:58:08.722400 18679 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 FOLLOWER]: Leader election won for term 3
I20250904 01:58:08.723179 18679 raft_consensus.cc:695] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [term 3 LEADER]: Becoming Leader. State: Replica: cac61d83403b4e51875bc24a3a91a3eb, State: Running, Role: LEADER
I20250904 01:58:08.722846 18120 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac61d83403b4e51875bc24a3a91a3eb in term 3.
I20250904 01:58:08.724546 18679 consensus_queue.cc:237] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:08.730859 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:08.731825 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:08.739738 18696 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:08.751330 18696 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:08.757234 18696 catalog_manager.cc:1261] Loaded cluster ID: 360389147c5748c9925cf97041f6ced8
I20250904 01:58:08.757577 18696 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:08.764539 18696 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:08.770228 18696 catalog_manager.cc:5966] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb: Loaded TSK: 0
I20250904 01:58:08.771888 18696 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:08.777375 18696 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:08.782580 18696 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:58:09.224798 18120 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Refusing update from remote peer cac61d83403b4e51875bc24a3a91a3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 01:58:09.230620 18679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:58:09.262567 18697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.263854 18697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:09.287845 18697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.288658 18697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:09.290457 18277 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Refusing update from remote peer cac61d83403b4e51875bc24a3a91a3eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 01:58:09.297765 18699 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.299876 18699 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:09.310124 18703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:58:09.313308 18520 heartbeater.cc:499] Master 127.16.112.189:38643 was elected leader, sending a full tablet report...
I20250904 01:58:09.356199 18706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac61d83403b4e51875bc24a3a91a3eb. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.357235 18706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:09.356791 18709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.357851 18709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:09.572679 18665 heartbeater.cc:499] Master 127.16.112.189:38643 was elected leader, sending a full tablet report...
I20250904 01:58:09.781606 18697 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:09.782104 18697 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:09.783887 18697 leader_election.cc:290] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643), 4904879f942d4b9f8a9fb8f5de6cfbe6 (127.16.112.188:35271)
I20250904 01:58:09.786528 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:09.788934 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" is_pre_election: true
I20250904 01:58:09.790643 18098 leader_election.cc:304] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5748fb924edd4e1b8a52330cbe5d3d5b; no voters: 4904879f942d4b9f8a9fb8f5de6cfbe6, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:09.791788 18697 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5748fb924edd4e1b8a52330cbe5d3d5b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250904 01:58:08.436451 18686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:09.822350 18688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:09.825146 18687 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1383 milliseconds
W20250904 01:58:09.826195 18676 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.390s user 0.412s sys 0.869s
W20250904 01:58:09.826548 18676 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.391s user 0.412s sys 0.870s
I20250904 01:58:09.826841 18676 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:09.826776 18703 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cac61d83403b4e51875bc24a3a91a3eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } } }
I20250904 01:58:09.827682 18703 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac61d83403b4e51875bc24a3a91a3eb [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:09.828390 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:09.829038 18709 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cac61d83403b4e51875bc24a3a91a3eb)
I20250904 01:58:09.829640 18709 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 46159 } } peers { permanent_uuid: "cac61d83403b4e51875bc24a3a91a3eb" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38643 } } peers { permanent_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 35271 } }
I20250904 01:58:09.831720 18709 leader_election.cc:290] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5748fb924edd4e1b8a52330cbe5d3d5b (127.16.112.190:46159), cac61d83403b4e51875bc24a3a91a3eb (127.16.112.189:38643)
I20250904 01:58:09.838474 18197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cac61d83403b4e51875bc24a3a91a3eb" is_pre_election: true
I20250904 01:58:09.840066 18120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4904879f942d4b9f8a9fb8f5de6cfbe6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5748fb924edd4e1b8a52330cbe5d3d5b" is_pre_election: true
I20250904 01:58:09.843287 18253 leader_election.cc:304] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4904879f942d4b9f8a9fb8f5de6cfbe6; no voters: 5748fb924edd4e1b8a52330cbe5d3d5b, cac61d83403b4e51875bc24a3a91a3eb
I20250904 01:58:09.846760 18709 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4904879f942d4b9f8a9fb8f5de6cfbe6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:09.848664 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:09.850226 18676 hybrid_clock.cc:648] HybridClock initialized: now 1756951089850162 us; error 113 us; skew 500 ppm
I20250904 01:58:09.851416 18676 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:09.860072 18676 webserver.cc:480] Webserver started at http://127.16.112.131:35487/ using document root <none> and password file <none>
I20250904 01:58:09.861352 18676 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:09.861667 18676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:09.862248 18676 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:09.869616 18676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/instance:
uuid: "7c777df1b953462db1d7d8631446a111"
format_stamp: "Formatted at 2025-09-04 01:58:09 on dist-test-slave-pl7p"
I20250904 01:58:09.871203 18676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance:
uuid: "7c777df1b953462db1d7d8631446a111"
format_stamp: "Formatted at 2025-09-04 01:58:09 on dist-test-slave-pl7p"
I20250904 01:58:09.880645 18676 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20250904 01:58:09.886576 18717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:09.887576 18676 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250904 01:58:09.887867 18676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
uuid: "7c777df1b953462db1d7d8631446a111"
format_stamp: "Formatted at 2025-09-04 01:58:09 on dist-test-slave-pl7p"
I20250904 01:58:09.888164 18676 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:09.950551 18676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:09.951939 18676 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:09.952354 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:09.954830 18676 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:09.958738 18676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:09.958942 18676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:09.959198 18676 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:09.959357 18676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:10.093307 18676 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.131:45377
I20250904 01:58:10.093451 18829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.131:45377 every 8 connection(s)
I20250904 01:58:10.096124 18676 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
I20250904 01:58:10.100431 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 18676
I20250904 01:58:10.101016 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance
I20250904 01:58:10.149397 18831 heartbeater.cc:344] Connected to a master server at 127.16.112.188:35271
I20250904 01:58:10.150106 18831 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:10.151697 18831 heartbeater.cc:507] Master 127.16.112.188:35271 requested a full tablet report, sending...
I20250904 01:58:10.155206 18268 ts_manager.cc:194] Registered new tserver with Master: 7c777df1b953462db1d7d8631446a111 (127.16.112.131:45377)
I20250904 01:58:10.166002 18832 heartbeater.cc:344] Connected to a master server at 127.16.112.190:46159
I20250904 01:58:10.166574 18832 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:10.166960 18830 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38643
I20250904 01:58:10.167310 18830 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:10.167425 18832 heartbeater.cc:507] Master 127.16.112.190:46159 requested a full tablet report, sending...
I20250904 01:58:10.168157 18830 heartbeater.cc:507] Master 127.16.112.189:38643 requested a full tablet report, sending...
I20250904 01:58:10.170177 18186 ts_manager.cc:194] Registered new tserver with Master: 7c777df1b953462db1d7d8631446a111 (127.16.112.131:45377)
I20250904 01:58:10.170503 18112 ts_manager.cc:194] Registered new tserver with Master: 7c777df1b953462db1d7d8631446a111 (127.16.112.131:45377)
I20250904 01:58:10.172361 18186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.131:41139
I20250904 01:58:10.186981 16834 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master_failover-itest.cc:235: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250904 01:58:10.221333 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18356
I20250904 01:58:10.243562 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18524
I20250904 01:58:10.267143 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18676
I20250904 01:58:10.286583 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18080
I20250904 01:58:10.317528 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18149
I20250904 01:58:10.343405 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 18223
Shutting down hive metastore.
2025-09-04T01:58:10Z chronyd exiting
[ SKIPPED ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1 (28123 ms)
[ RUN ] HmsConfigurations/MasterFailoverTest.TestKUDU1374/1
2025-09-04T01:58:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T01:58:10Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/security.properties
2025-09-04 01:58:16: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250904 01:58:24.429735 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:40807
--webserver_interface=127.16.112.190
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--hive_metastore_uris=thrift://127.0.0.1:37549
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
W20250904 01:58:24.710254 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:24.710781 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:24.711170 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:24.741240 19089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:58:24.741514 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:24.741755 19089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:58:24.741958 19089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:58:24.775447 19089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37549
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.112.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:24.776700 19089 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:24.778216 19089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:24.788235 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:24.789294 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:24.792918 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:25.954201 19097 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1160 milliseconds
I20250904 01:58:25.954303 19089 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:25.955404 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:25.958346 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:25.959779 19089 hybrid_clock.cc:648] HybridClock initialized: now 1756951105959742 us; error 56 us; skew 500 ppm
I20250904 01:58:25.960462 19089 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:25.966768 19089 webserver.cc:480] Webserver started at http://127.16.112.190:35885/ using document root <none> and password file <none>
I20250904 01:58:25.967634 19089 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:25.967814 19089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:25.968194 19089 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:25.972328 19089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data/instance:
uuid: "a616982f94594b49be0986767f8616c1"
format_stamp: "Formatted at 2025-09-04 01:58:25 on dist-test-slave-pl7p"
I20250904 01:58:25.973320 19089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance:
uuid: "a616982f94594b49be0986767f8616c1"
format_stamp: "Formatted at 2025-09-04 01:58:25 on dist-test-slave-pl7p"
I20250904 01:58:25.980226 19089 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250904 01:58:25.985508 19105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:25.986469 19089 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250904 01:58:25.986757 19089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "a616982f94594b49be0986767f8616c1"
format_stamp: "Formatted at 2025-09-04 01:58:25 on dist-test-slave-pl7p"
I20250904 01:58:25.987038 19089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:26.055871 19089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:26.057212 19089 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:26.057605 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:26.123108 19089 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.190:40807
I20250904 01:58:26.123193 19156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.190:40807 every 8 connection(s)
I20250904 01:58:26.125671 19089 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
I20250904 01:58:26.126879 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19089
I20250904 01:58:26.127385 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance
I20250904 01:58:26.129516 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:38857
--webserver_interface=127.16.112.189
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--hive_metastore_uris=thrift://127.0.0.1:37549
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:58:26.130458 19157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:26.139825 19157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:26.161633 19157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:26.163762 19109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
W20250904 01:58:26.167397 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:26.216814 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 1
W20250904 01:58:26.220908 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:26.289254 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 2
W20250904 01:58:26.293150 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:26.384514 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 3
W20250904 01:58:26.388381 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
W20250904 01:58:26.408952 19158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:26.409513 19158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:26.409966 19158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:26.438953 19158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:58:26.439256 19158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:26.439502 19158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:58:26.439749 19158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:58:26.472931 19158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37549
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.112.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:26.474215 19158 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:26.475786 19158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:26.486341 19167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:26.531769 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 4
W20250904 01:58:26.538671 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:26.838193 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 5
W20250904 01:58:26.845715 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:27.393277 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 6
W20250904 01:58:27.424458 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:38857: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
W20250904 01:58:27.890040 19166 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 19158
W20250904 01:58:28.005573 19158 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.519s user 0.456s sys 1.058s
W20250904 01:58:28.005988 19158 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.520s user 0.457s sys 1.058s
W20250904 01:58:26.486799 19168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:28.007754 19170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:28.007817 19169 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1521 milliseconds
I20250904 01:58:28.007751 19158 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:28.011516 19158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:28.013767 19158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:28.015086 19158 hybrid_clock.cc:648] HybridClock initialized: now 1756951108015046 us; error 45 us; skew 500 ppm
I20250904 01:58:28.015859 19158 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:28.021596 19158 webserver.cc:480] Webserver started at http://127.16.112.189:44993/ using document root <none> and password file <none>
I20250904 01:58:28.022420 19158 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:28.022630 19158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:28.023066 19158 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:28.027271 19158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data/instance:
uuid: "c948e45d03324d00b72a04b2848e8a41"
format_stamp: "Formatted at 2025-09-04 01:58:28 on dist-test-slave-pl7p"
I20250904 01:58:28.028263 19158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance:
uuid: "c948e45d03324d00b72a04b2848e8a41"
format_stamp: "Formatted at 2025-09-04 01:58:28 on dist-test-slave-pl7p"
I20250904 01:58:28.034603 19158 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250904 01:58:28.039629 19179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:28.040517 19158 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250904 01:58:28.040808 19158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal
uuid: "c948e45d03324d00b72a04b2848e8a41"
format_stamp: "Formatted at 2025-09-04 01:58:28 on dist-test-slave-pl7p"
I20250904 01:58:28.041090 19158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:28.091236 19158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:28.092594 19158 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:28.093009 19158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:28.161974 19158 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.189:38857
I20250904 01:58:28.162029 19230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.189:38857 every 8 connection(s)
I20250904 01:58:28.164531 19158 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
I20250904 01:58:28.168910 19231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:28.170668 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19158
I20250904 01:58:28.171034 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance
I20250904 01:58:28.173055 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:38591
--webserver_interface=127.16.112.188
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--hive_metastore_uris=thrift://127.0.0.1:37549
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:58:28.178807 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:28.202174 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:28.214660 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:28.216955 19180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
W20250904 01:58:28.220964 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.270370 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 1
W20250904 01:58:28.274298 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.342751 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 2
W20250904 01:58:28.346676 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.438015 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 3
W20250904 01:58:28.441807 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
W20250904 01:58:28.456038 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:28.456573 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:28.457038 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250904 01:58:28.485005 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } attempt: 7
W20250904 01:58:28.486254 19232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:58:28.486752 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:28.487205 19232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:58:28.487506 19232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:58:28.495326 19157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:28.500028 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.524335 19232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37549
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.112.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:28.525705 19232 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:28.527274 19232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:28.538190 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:28.558372 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 1
W20250904 01:58:28.565352 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.585197 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 4
W20250904 01:58:28.591689 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.646867 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 2
W20250904 01:58:28.653687 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.739223 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 3
W20250904 01:58:28.745988 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:28.886566 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 4
I20250904 01:58:28.891145 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 5
W20250904 01:58:28.897640 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
W20250904 01:58:28.900298 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:29.183809 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 5
W20250904 01:58:29.189124 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:29.445142 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 6
W20250904 01:58:29.471812 19231 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
I20250904 01:58:29.741596 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 6
W20250904 01:58:29.767297 19157 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:38591: Network error: Client connection negotiation failed: client connection to 127.16.112.188:38591: connect: Connection refused (error 111)
W20250904 01:58:29.941354 19243 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 19232
W20250904 01:58:28.539120 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:30.245991 19232 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.708s user 0.568s sys 1.137s
W20250904 01:58:30.246423 19232 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.708s user 0.568s sys 1.137s
W20250904 01:58:30.248308 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:30.250167 19246 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1708 milliseconds
I20250904 01:58:30.250183 19232 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:30.251369 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:30.253513 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:30.254870 19232 hybrid_clock.cc:648] HybridClock initialized: now 1756951110254835 us; error 40 us; skew 500 ppm
I20250904 01:58:30.255600 19232 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:30.260960 19232 webserver.cc:480] Webserver started at http://127.16.112.188:40843/ using document root <none> and password file <none>
I20250904 01:58:30.261898 19232 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:30.262094 19232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:30.262495 19232 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:30.266602 19232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data/instance:
uuid: "9dae92e829f24e178cf3cb3479bf09b7"
format_stamp: "Formatted at 2025-09-04 01:58:30 on dist-test-slave-pl7p"
I20250904 01:58:30.267534 19232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance:
uuid: "9dae92e829f24e178cf3cb3479bf09b7"
format_stamp: "Formatted at 2025-09-04 01:58:30 on dist-test-slave-pl7p"
I20250904 01:58:30.273981 19232 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250904 01:58:30.278875 19258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:30.279685 19232 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250904 01:58:30.279968 19232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal
uuid: "9dae92e829f24e178cf3cb3479bf09b7"
format_stamp: "Formatted at 2025-09-04 01:58:30 on dist-test-slave-pl7p"
I20250904 01:58:30.280246 19232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:30.339779 19232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:30.341176 19232 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:30.341607 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:30.406395 19232 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.188:38591
I20250904 01:58:30.406450 19309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.188:38591 every 8 connection(s)
I20250904 01:58:30.408813 19232 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
I20250904 01:58:30.413230 19310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:30.414608 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19232
I20250904 01:58:30.414970 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance
I20250904 01:58:30.427445 19310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:30.453781 19310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:30.461966 19310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:30.477205 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: Bootstrap starting.
I20250904 01:58:30.482510 19310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:30.484517 19310 log.cc:826] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:30.488843 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: No bootstrap required, opened a new log
I20250904 01:58:30.506206 19310 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.506783 19310 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:30.507027 19310 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dae92e829f24e178cf3cb3479bf09b7, State: Initialized, Role: FOLLOWER
I20250904 01:58:30.507637 19310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.509666 19318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.510181 19318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:30.511159 19310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:58:30.532430 19231 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 7
I20250904 01:58:30.535718 19232 master_runner.cc:186] Error getting master registration for 127.16.112.190:40807: OK, instance_id { permanent_uuid: "a616982f94594b49be0986767f8616c1" instance_seqno: 1756951106105423 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20250904 01:58:30.562307 19332 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250904 01:58:30.562546 19232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:58:30.562628 19332 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:58:30.574622 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: Bootstrap starting.
I20250904 01:58:30.581828 19231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:30.584230 19231 log.cc:826] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: Log is configured to *not* fsync() on all Append() calls
W20250904 01:58:30.596439 19264 tablet.cc:2378] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:58:30.609033 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: No bootstrap required, opened a new log
I20250904 01:58:30.633230 19231 raft_consensus.cc:357] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.634264 19231 raft_consensus.cc:383] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:30.634616 19231 raft_consensus.cc:738] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c948e45d03324d00b72a04b2848e8a41, State: Initialized, Role: FOLLOWER
I20250904 01:58:30.635571 19231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.640933 19337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.641681 19337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:30.642593 19231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:58:30.670854 19337 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:30.671419 19337 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.675701 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:30.676478 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 0.
I20250904 01:58:30.677906 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:30.687768 19337 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:30.688526 19337 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:58:30.689298 19337 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:30.690037 19337 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:30.673496 19158 master_runner.cc:186] Error getting master registration for 127.16.112.190:40807: OK, instance_id { permanent_uuid: "a616982f94594b49be0986767f8616c1" instance_seqno: 1756951106105423 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:58:30.703491 19158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:58:30.707675 19352 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:58:30.708007 19352 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:58:30.710350 19337 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.713917 19337 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 1 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:30.716113 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7"
I20250904 01:58:30.716598 19285 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:30.727377 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 1.
I20250904 01:58:30.728339 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:30.729202 19337 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:58:30.729725 19337 raft_consensus.cc:695] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 LEADER]: Becoming Leader. State: Replica: c948e45d03324d00b72a04b2848e8a41, State: Running, Role: LEADER
I20250904 01:58:30.730521 19337 consensus_queue.cc:237] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.740221 19353 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.740895 19353 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:30.743474 19356 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:30.753012 19356 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:30.771690 19285 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Refusing update from remote peer c948e45d03324d00b72a04b2848e8a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:58:30.773419 19353 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:58:30.800758 19157 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } attempt: 7
I20250904 01:58:30.817901 19357 mvcc.cc:204] Tried to move back new op lower bound from 7196471749698908160 to 7196471749560897536. Current Snapshot: MvccSnapshot[applied={T|T < 7196471749698908160}]
I20250904 01:58:30.818336 19318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.819010 19318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:30.835858 19358 mvcc.cc:204] Tried to move back new op lower bound from 7196471749698908160 to 7196471749560897536. Current Snapshot: MvccSnapshot[applied={T|T < 7196471749698908160}]
I20250904 01:58:30.837554 19318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.840143 19353 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.840590 19318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:30.840751 19353 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:30.844621 19356 catalog_manager.cc:1349] Generated new cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:30.844904 19356 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:30.859978 19157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: Bootstrap starting.
I20250904 01:58:30.872277 19157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:30.881850 19157 log.cc:826] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:30.891029 19356 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 01:58:30.892890 19356 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:30.909318 19157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: No bootstrap required, opened a new log
I20250904 01:58:30.950560 19157 raft_consensus.cc:357] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.951483 19157 raft_consensus.cc:383] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:30.951802 19157 raft_consensus.cc:738] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a616982f94594b49be0986767f8616c1, State: Initialized, Role: FOLLOWER
I20250904 01:58:30.954104 19157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:30.960304 19356 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: Generated new TSK 0
I20250904 01:58:30.961385 19356 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:30.966154 19356 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:30.970311 19356 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:58:30.998497 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:30.999217 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:31.000648 19157 sys_catalog.cc:564] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:58:31.000833 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:31.001962 19130 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:31.002419 19132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:31.012404 19130 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Refusing update from remote peer c948e45d03324d00b72a04b2848e8a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:58:31.014201 19365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250904 01:58:31.028223 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.129:0
--local_ip_for_outbound_sockets=127.16.112.129
--webserver_interface=127.16.112.129
--webserver_port=0
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:31.078656 19376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:31.080654 19376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:31.079907 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:31.081225 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:31.135721 19386 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250904 01:58:31.436944 19365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:31.437681 19365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
W20250904 01:58:31.541254 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:31.541867 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:31.542768 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250904 01:58:31.568550 19332 catalog_manager.cc:1261] Loaded cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:31.568918 19332 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: loading cluster ID for follower catalog manager: success
I20250904 01:58:31.576804 19332 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: acquiring CA information for follower catalog manager: success
I20250904 01:58:31.581933 19332 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 01:58:31.596035 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:31.597499 19372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.129
I20250904 01:58:31.656898 19372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.112.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.129
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:31.658502 19372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:31.660466 19372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:31.674686 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:31.950551 19398 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:31.951566 19398 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:31.955145 19398 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:31.956723 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:31.958992 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:31.960888 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:31.962244 19398 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:32.172006 19386 catalog_manager.cc:1261] Loaded cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:32.185801 19386 catalog_manager.cc:1554] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: loading cluster ID for follower catalog manager: success
I20250904 01:58:32.220520 19386 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: acquiring CA information for follower catalog manager: success
I20250904 01:58:32.252926 19386 catalog_manager.cc:1604] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:58:32.397701 19400 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:32.398483 19400 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:32.414656 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:32.421362 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:32.423318 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:32.426610 19400 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:32.426887 19401 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:32.515344 19402 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:32.525020 19402 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:32.528230 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:32.534454 19402 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:32.534752 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:32.536527 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:32.537726 19402 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:33.119014 19403 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:33.121721 19403 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:33.126257 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:33.128093 19403 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:33.129105 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:33.130578 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:33.131374 19403 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:58:33.139550 19153 debug-util.cc:398] Leaking SignalData structure 0x7b080007d7c0 after lost signal to thread 19090
W20250904 01:58:33.140308 19153 debug-util.cc:398] Leaking SignalData structure 0x7b080007dac0 after lost signal to thread 19156
W20250904 01:58:31.677321 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:33.078591 19391 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 19372
W20250904 01:58:33.506454 19372 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.828s user 0.000s sys 0.002s
W20250904 01:58:33.514510 19372 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.836s user 0.000s sys 0.003s
W20250904 01:58:33.507063 19394 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1826 milliseconds
I20250904 01:58:33.516263 19372 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 01:58:33.516786 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:33.518280 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:33.530099 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:33.531567 19372 hybrid_clock.cc:648] HybridClock initialized: now 1756951113531526 us; error 44 us; skew 500 ppm
I20250904 01:58:33.532670 19372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:33.557329 19372 webserver.cc:480] Webserver started at http://127.16.112.129:38153/ using document root <none> and password file <none>
I20250904 01:58:33.558622 19372 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:33.558914 19372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:33.559479 19372 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:33.576000 19372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data/instance:
uuid: "68d1e8697f2a436880371e195ef692e4"
format_stamp: "Formatted at 2025-09-04 01:58:33 on dist-test-slave-pl7p"
I20250904 01:58:33.577487 19372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance:
uuid: "68d1e8697f2a436880371e195ef692e4"
format_stamp: "Formatted at 2025-09-04 01:58:33 on dist-test-slave-pl7p"
I20250904 01:58:33.601418 19372 fs_manager.cc:696] Time spent creating directory manager: real 0.023s user 0.009s sys 0.001s
I20250904 01:58:33.627095 19408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:33.628307 19372 fs_manager.cc:730] Time spent opening block manager: real 0.022s user 0.002s sys 0.003s
I20250904 01:58:33.628711 19372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
uuid: "68d1e8697f2a436880371e195ef692e4"
format_stamp: "Formatted at 2025-09-04 01:58:33 on dist-test-slave-pl7p"
I20250904 01:58:33.629132 19372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:33.683528 19409 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:33.684166 19409 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:33.687031 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:33.689939 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:33.691344 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:33.696940 19409 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:33.697913 19409 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:33.857913 19372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:33.860013 19372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:33.860591 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:33.881091 19372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:33.900761 19372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:33.901084 19372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:33.901396 19372 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:33.901630 19372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.002s
I20250904 01:58:33.966228 19422 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:33.973881 19422 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:33.977655 19422 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:33.977676 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:33.979339 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:33.981316 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:33.982084 19422 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:34.548439 19511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:34.549172 19511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:34.552590 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:34.555953 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:34.557848 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:34.563299 19511 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:34.564349 19511 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:34.589277 19372 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.129:38241
I20250904 01:58:34.589368 19526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.129:38241 every 8 connection(s)
I20250904 01:58:34.591668 19372 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
I20250904 01:58:34.594785 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19372
I20250904 01:58:34.595664 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance
I20250904 01:58:34.618971 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.130:0
--local_ip_for_outbound_sockets=127.16.112.130
--webserver_interface=127.16.112.130
--webserver_port=0
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:34.702047 19533 heartbeater.cc:344] Connected to a master server at 127.16.112.190:40807
I20250904 01:58:34.702517 19533 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:34.703676 19533 heartbeater.cc:507] Master 127.16.112.190:40807 requested a full tablet report, sending...
I20250904 01:58:34.707501 19122 ts_manager.cc:194] Registered new tserver with Master: 68d1e8697f2a436880371e195ef692e4 (127.16.112.129:38241)
I20250904 01:58:34.710819 19527 heartbeater.cc:344] Connected to a master server at 127.16.112.188:38591
I20250904 01:58:34.711163 19527 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:34.711787 19528 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38857
I20250904 01:58:34.712112 19528 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:34.712075 19527 heartbeater.cc:507] Master 127.16.112.188:38591 requested a full tablet report, sending...
I20250904 01:58:34.713066 19528 heartbeater.cc:507] Master 127.16.112.189:38857 requested a full tablet report, sending...
I20250904 01:58:34.715332 19275 ts_manager.cc:194] Registered new tserver with Master: 68d1e8697f2a436880371e195ef692e4 (127.16.112.129:38241)
I20250904 01:58:34.715474 19196 ts_manager.cc:194] Registered new tserver with Master: 68d1e8697f2a436880371e195ef692e4 (127.16.112.129:38241)
I20250904 01:58:34.718050 19196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.129:37831
I20250904 01:58:34.723542 19539 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:34.724237 19539 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:34.726729 19539 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:34.727409 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:34.729316 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:34.731822 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:34.733112 19539 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:58:34.999236 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:34.999888 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:35.000801 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:35.054013 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:35.055593 19537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.130
I20250904 01:58:35.116091 19537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.112.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.130
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:35.117836 19537 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:35.119993 19537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:35.134729 19545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:35.180395 19550 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:35.181936 19550 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:35.188562 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:35.195487 19550 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:35.195425 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:35.197405 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:35.201757 19550 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:35.222684 19528 heartbeater.cc:499] Master 127.16.112.189:38857 was elected leader, sending a full tablet report...
I20250904 01:58:35.781246 19552 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:35.782878 19552 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:35.786450 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:35.788789 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:35.789144 19552 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:35.790462 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:35.791483 19552 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:36.256202 19553 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:36.257490 19553 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.283872 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:36.285135 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:36.285370 19553 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:36.288828 19109 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:36.296337 19553 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:36.314257 19554 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:36.315014 19554 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.317135 19554 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:36.318351 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:36.322299 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:36.324513 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:36.325635 19554 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:36.838274 19557 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:36.845858 19557 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.849329 19557 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:36.851596 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:36.849139 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:36.853487 19131 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dae92e829f24e178cf3cb3479bf09b7 in term 1.
I20250904 01:58:36.854660 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1; no voters:
I20250904 01:58:36.855377 19557 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:58:36.855691 19557 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:36.856021 19557 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:58:36.864728 19557 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.868242 19556 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:36.869784 19556 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.874909 19557 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:36.876350 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41"
I20250904 01:58:36.878548 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:36.879932 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:36.880968 19557 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250904 01:58:36.883729 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:36.884454 19285 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a616982f94594b49be0986767f8616c1 in current term 2: Already voted for candidate 9dae92e829f24e178cf3cb3479bf09b7 in this term.
I20250904 01:58:36.886770 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:36.888237 19109 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:36.889164 19556 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:36.889675 19556 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 1 FOLLOWER]: Advancing to term 2
I20250904 01:58:36.897262 19556 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:58:36.935180 19285 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Rejecting Update request from peer c948e45d03324d00b72a04b2848e8a41 for earlier term 1. Current term is 2. Ops: []
I20250904 01:58:36.936791 19558 consensus_queue.cc:1046] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:58:36.946609 19560 raft_consensus.cc:3053] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 LEADER]: Stepping down as leader of term 1
I20250904 01:58:36.947041 19560 raft_consensus.cc:738] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c948e45d03324d00b72a04b2848e8a41, State: Running, Role: LEADER
I20250904 01:58:36.947916 19560 consensus_queue.cc:260] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:36.949323 19560 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 1 FOLLOWER]: Advancing to term 2
W20250904 01:58:36.536084 19544 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 19537
W20250904 01:58:37.030300 19537 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.893s user 0.484s sys 0.989s
W20250904 01:58:37.030747 19537 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.894s user 0.484s sys 0.989s
W20250904 01:58:35.139982 19546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:37.032320 19548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:37.035272 19547 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1894 milliseconds
I20250904 01:58:37.035328 19537 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:37.036412 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:37.038406 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:37.039760 19537 hybrid_clock.cc:648] HybridClock initialized: now 1756951117039727 us; error 32 us; skew 500 ppm
I20250904 01:58:37.040474 19537 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:37.046305 19537 webserver.cc:480] Webserver started at http://127.16.112.130:43569/ using document root <none> and password file <none>
I20250904 01:58:37.047225 19537 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:37.047417 19537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:37.047806 19537 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:37.051800 19537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data/instance:
uuid: "99367d2d849740cfb2c8275cf5c932ba"
format_stamp: "Formatted at 2025-09-04 01:58:37 on dist-test-slave-pl7p"
I20250904 01:58:37.052826 19537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance:
uuid: "99367d2d849740cfb2c8275cf5c932ba"
format_stamp: "Formatted at 2025-09-04 01:58:37 on dist-test-slave-pl7p"
I20250904 01:58:37.059365 19537 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20250904 01:58:37.064668 19565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:37.065598 19537 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250904 01:58:37.065910 19537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
uuid: "99367d2d849740cfb2c8275cf5c932ba"
format_stamp: "Formatted at 2025-09-04 01:58:37 on dist-test-slave-pl7p"
I20250904 01:58:37.066205 19537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:37.120771 19537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:37.122177 19537 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:37.122633 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:37.125111 19537 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:37.129236 19537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:37.129465 19537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:37.129742 19537 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:37.129928 19537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:37.268627 19537 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.130:35497
I20250904 01:58:37.268775 19677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.130:35497 every 8 connection(s)
I20250904 01:58:37.270992 19537 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
I20250904 01:58:37.279302 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19537
I20250904 01:58:37.279870 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance
I20250904 01:58:37.313658 19680 heartbeater.cc:344] Connected to a master server at 127.16.112.190:40807
I20250904 01:58:37.314186 19680 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:37.315471 19680 heartbeater.cc:507] Master 127.16.112.190:40807 requested a full tablet report, sending...
I20250904 01:58:37.316509 19678 heartbeater.cc:344] Connected to a master server at 127.16.112.188:38591
I20250904 01:58:37.316637 19679 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38857
I20250904 01:58:37.316895 19678 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:37.316967 19679 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:37.317852 19678 heartbeater.cc:507] Master 127.16.112.188:38591 requested a full tablet report, sending...
I20250904 01:58:37.317898 19679 heartbeater.cc:507] Master 127.16.112.189:38857 requested a full tablet report, sending...
I20250904 01:58:37.318608 19122 ts_manager.cc:194] Registered new tserver with Master: 99367d2d849740cfb2c8275cf5c932ba (127.16.112.130:35497)
I20250904 01:58:37.320662 19274 ts_manager.cc:194] Registered new tserver with Master: 99367d2d849740cfb2c8275cf5c932ba (127.16.112.130:35497)
I20250904 01:58:37.321794 19195 ts_manager.cc:194] Registered new tserver with Master: 99367d2d849740cfb2c8275cf5c932ba (127.16.112.130:35497)
I20250904 01:58:37.492125 19688 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:37.492475 19688 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:37.493810 19688 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:37.494516 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:37.494654 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:37.495179 19206 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a616982f94594b49be0986767f8616c1 in term 2.
I20250904 01:58:37.495232 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a616982f94594b49be0986767f8616c1 in term 2.
I20250904 01:58:37.496064 19109 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:37.496693 19688 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 01:58:37.496906 19688 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:37.497138 19688 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:58:37.502673 19688 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:37.504489 19688 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 3 election: Requested vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:37.505326 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41"
I20250904 01:58:37.505841 19206 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:58:37.506562 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7"
I20250904 01:58:37.507218 19285 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:58:37.511485 19206 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a616982f94594b49be0986767f8616c1 in term 3.
I20250904 01:58:37.512253 19109 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:37.512358 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a616982f94594b49be0986767f8616c1 in term 3.
I20250904 01:58:37.512859 19688 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 01:58:37.513897 19688 raft_consensus.cc:695] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 LEADER]: Becoming Leader. State: Replica: a616982f94594b49be0986767f8616c1, State: Running, Role: LEADER
I20250904 01:58:37.514535 19688 consensus_queue.cc:237] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:37.518903 19689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a616982f94594b49be0986767f8616c1. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:37.519455 19689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:37.520728 19691 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:37.526337 19691 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:37.529690 19691 catalog_manager.cc:1261] Loaded cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:37.529847 19691 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:37.532645 19691 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:37.535070 19691 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1: Loaded TSK: 0
I20250904 01:58:37.536031 19691 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:37.538429 19691 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:37.540634 19691 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:58:37.551594 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.131:0
--local_ip_for_outbound_sockets=127.16.112.131
--webserver_interface=127.16.112.131
--webserver_port=0
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--builtin_ntp_servers=127.16.112.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:58:37.724313 19352 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:58:37.824182 19122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.130:53179
W20250904 01:58:37.838450 19692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:37.838975 19692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:37.839579 19692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:37.869001 19692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:37.869985 19692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.131
I20250904 01:58:37.903041 19692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.112.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:40807,127.16.112.189:38857,127.16.112.188:38591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.131
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:37.904239 19692 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:37.905752 19692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:37.916278 19698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:37.924427 19206 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Refusing update from remote peer a616982f94594b49be0986767f8616c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 01:58:37.926523 19689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:58:37.950265 19703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a616982f94594b49be0986767f8616c1. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:37.960939 19703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:37.968961 19703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:37.969864 19703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:37.974412 19689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:37.975325 19285 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Refusing update from remote peer a616982f94594b49be0986767f8616c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 01:58:37.975425 19689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:37.977522 19688 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250904 01:58:38.008975 19708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a616982f94594b49be0986767f8616c1. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:38.009691 19708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:38.011142 19711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:38.012236 19711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:38.251849 19533 heartbeater.cc:499] Master 127.16.112.190:40807 was elected leader, sending a full tablet report...
I20250904 01:58:38.327723 19680 heartbeater.cc:499] Master 127.16.112.190:40807 was elected leader, sending a full tablet report...
I20250904 01:58:38.456427 19703 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:38.456957 19703 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:38.459363 19703 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:38.461199 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:38.463748 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:38.466275 19182 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c948e45d03324d00b72a04b2848e8a41; no voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1
I20250904 01:58:38.467424 19703 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:38.514115 19708 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:38.515142 19708 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:38.519202 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:38.523111 19205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:38.524827 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:38.525774 19708 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:38.526522 19708 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:38.563230 19688 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a616982f94594b49be0986767f8616c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:38.564205 19688 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:39.084905 19716 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:39.085819 19716 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:39.089886 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:39.099717 19205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:39.102039 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:39.107425 19716 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:39.111023 19716 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250904 01:58:37.917809 19699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:39.451785 19692 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.535s user 0.485s sys 1.005s
W20250904 01:58:39.452301 19692 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.535s user 0.485s sys 1.005s
W20250904 01:58:39.319391 19697 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 19692
I20250904 01:58:39.453133 19718 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
W20250904 01:58:39.453972 19701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:58:39.453812 19718 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:39.456564 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:39.457278 19718 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:39.457963 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
W20250904 01:58:39.458940 19700 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1541 milliseconds
I20250904 01:58:39.459007 19692 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:39.459707 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c948e45d03324d00b72a04b2848e8a41; no voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1
I20250904 01:58:39.460415 19718 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:39.460605 19692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:39.462723 19692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:39.464069 19692 hybrid_clock.cc:648] HybridClock initialized: now 1756951119464022 us; error 49 us; skew 500 ppm
I20250904 01:58:39.464813 19692 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:39.471413 19692 webserver.cc:480] Webserver started at http://127.16.112.131:43575/ using document root <none> and password file <none>
I20250904 01:58:39.472339 19692 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:39.472554 19692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:39.472997 19692 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:39.477247 19692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data/instance:
uuid: "91192ab5da56454c9a8ded8e5d24b782"
format_stamp: "Formatted at 2025-09-04 01:58:39 on dist-test-slave-pl7p"
I20250904 01:58:39.478335 19692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance:
uuid: "91192ab5da56454c9a8ded8e5d24b782"
format_stamp: "Formatted at 2025-09-04 01:58:39 on dist-test-slave-pl7p"
I20250904 01:58:39.485110 19692 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250904 01:58:39.490598 19723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:39.491545 19692 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250904 01:58:39.491854 19692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
uuid: "91192ab5da56454c9a8ded8e5d24b782"
format_stamp: "Formatted at 2025-09-04 01:58:39 on dist-test-slave-pl7p"
I20250904 01:58:39.492175 19692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:39.564438 19692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:39.565946 19692 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:39.566375 19692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:39.568851 19692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:58:39.572994 19692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:58:39.573210 19692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:39.573441 19692 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:58:39.573609 19692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250904 01:58:39.613291 19717 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:39.613914 19717 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:39.616778 19717 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:39.617132 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:39.617599 19205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:39.619113 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:39.619971 19717 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:39.711380 19692 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.131:35845
I20250904 01:58:39.711488 19835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.131:35845 every 8 connection(s)
I20250904 01:58:39.714722 19692 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
I20250904 01:58:39.720069 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 19692
I20250904 01:58:39.720515 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance
I20250904 01:58:39.773061 19836 heartbeater.cc:344] Connected to a master server at 127.16.112.188:38591
I20250904 01:58:39.773591 19836 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:39.774611 19836 heartbeater.cc:507] Master 127.16.112.188:38591 requested a full tablet report, sending...
I20250904 01:58:39.776284 19837 heartbeater.cc:344] Connected to a master server at 127.16.112.189:38857
I20250904 01:58:39.776597 19837 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:39.777506 19837 heartbeater.cc:507] Master 127.16.112.189:38857 requested a full tablet report, sending...
I20250904 01:58:39.777510 19275 ts_manager.cc:194] Registered new tserver with Master: 91192ab5da56454c9a8ded8e5d24b782 (127.16.112.131:35845)
I20250904 01:58:39.778625 19840 heartbeater.cc:344] Connected to a master server at 127.16.112.190:40807
I20250904 01:58:39.778935 19840 heartbeater.cc:461] Registering TS with master...
I20250904 01:58:39.779695 19840 heartbeater.cc:507] Master 127.16.112.190:40807 requested a full tablet report, sending...
I20250904 01:58:39.779937 19195 ts_manager.cc:194] Registered new tserver with Master: 91192ab5da56454c9a8ded8e5d24b782 (127.16.112.131:35845)
I20250904 01:58:39.782157 19122 ts_manager.cc:194] Registered new tserver with Master: 91192ab5da56454c9a8ded8e5d24b782 (127.16.112.131:35845)
I20250904 01:58:39.783722 19122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.131:35521
I20250904 01:58:39.790432 16834 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 01:58:39.830554 19122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47692:
name: "default.test_kudu_1374"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 01:58:39.891014 19122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.test_kudu_1374 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 01:58:40.231992 19853 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:40.232448 19853 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:40.235864 19853 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:40.236320 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:40.236523 19205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:40.237957 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:40.238955 19853 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 01:58:40.288442 19840 heartbeater.cc:499] Master 127.16.112.190:40807 was elected leader, sending a full tablet report...
I20250904 01:58:40.305305 19122 catalog_manager.cc:2149] created HMS catalog entry for table [id=b19e6651a2e2493e8e29e2236aa35d38]
I20250904 01:58:40.406791 19771 tablet_service.cc:1468] Processing CreateTablet for tablet b365aecdaa6b403abaccba90c6b322a7 (DEFAULT_TABLE table=default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:58:40.407150 19613 tablet_service.cc:1468] Processing CreateTablet for tablet b365aecdaa6b403abaccba90c6b322a7 (DEFAULT_TABLE table=default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:58:40.408804 19613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b365aecdaa6b403abaccba90c6b322a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:40.410676 19460 tablet_service.cc:1468] Processing CreateTablet for tablet b365aecdaa6b403abaccba90c6b322a7 (DEFAULT_TABLE table=default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 01:58:40.412364 19460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b365aecdaa6b403abaccba90c6b322a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:40.414153 19771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b365aecdaa6b403abaccba90c6b322a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:40.451845 19871 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Bootstrap starting.
I20250904 01:58:40.465369 19871 tablet_bootstrap.cc:654] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:40.475840 19871 log.cc:826] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:40.489255 19873 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Bootstrap starting.
I20250904 01:58:40.492584 19871 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: No bootstrap required, opened a new log
I20250904 01:58:40.493180 19871 ts_tablet_manager.cc:1397] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.005s
I20250904 01:58:40.496665 19873 tablet_bootstrap.cc:654] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:40.499197 19873 log.cc:826] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:40.500994 19872 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Bootstrap starting.
I20250904 01:58:40.507444 19872 tablet_bootstrap.cc:654] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Neither blocks nor log segments found. Creating new log.
I20250904 01:58:40.523535 19873 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: No bootstrap required, opened a new log
I20250904 01:58:40.524103 19873 ts_tablet_manager.cc:1397] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.009s
I20250904 01:58:40.525312 19871 raft_consensus.cc:357] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.526312 19871 raft_consensus.cc:383] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:40.526638 19871 raft_consensus.cc:738] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99367d2d849740cfb2c8275cf5c932ba, State: Initialized, Role: FOLLOWER
I20250904 01:58:40.526928 19872 log.cc:826] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Log is configured to *not* fsync() on all Append() calls
I20250904 01:58:40.527487 19871 consensus_queue.cc:260] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.540692 19872 tablet_bootstrap.cc:492] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: No bootstrap required, opened a new log
I20250904 01:58:40.541352 19872 ts_tablet_manager.cc:1397] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.000s
I20250904 01:58:40.543066 19871 ts_tablet_manager.cc:1428] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Time spent starting tablet: real 0.050s user 0.034s sys 0.011s
W20250904 01:58:40.547822 19685 tablet.cc:2378] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:58:40.562265 19873 raft_consensus.cc:357] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.563298 19873 raft_consensus.cc:383] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:40.563632 19873 raft_consensus.cc:738] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d1e8697f2a436880371e195ef692e4, State: Initialized, Role: FOLLOWER
I20250904 01:58:40.564649 19873 consensus_queue.cc:260] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.574393 19873 ts_tablet_manager.cc:1428] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Time spent starting tablet: real 0.050s user 0.028s sys 0.006s
I20250904 01:58:40.595588 19872 raft_consensus.cc:357] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.596529 19872 raft_consensus.cc:383] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:58:40.596897 19872 raft_consensus.cc:738] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91192ab5da56454c9a8ded8e5d24b782, State: Initialized, Role: FOLLOWER
I20250904 01:58:40.597856 19872 consensus_queue.cc:260] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.602061 19872 ts_tablet_manager.cc:1428] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Time spent starting tablet: real 0.060s user 0.034s sys 0.000s
W20250904 01:58:40.663195 19536 tablet.cc:2378] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:58:40.733105 19879 raft_consensus.cc:491] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:40.733542 19879 raft_consensus.cc:513] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.736188 19879 leader_election.cc:290] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68d1e8697f2a436880371e195ef692e4 (127.16.112.129:38241), 99367d2d849740cfb2c8275cf5c932ba (127.16.112.130:35497)
I20250904 01:58:40.748775 19480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b365aecdaa6b403abaccba90c6b322a7" candidate_uuid: "91192ab5da56454c9a8ded8e5d24b782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d1e8697f2a436880371e195ef692e4" is_pre_election: true
I20250904 01:58:40.748971 19633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b365aecdaa6b403abaccba90c6b322a7" candidate_uuid: "91192ab5da56454c9a8ded8e5d24b782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99367d2d849740cfb2c8275cf5c932ba" is_pre_election: true
I20250904 01:58:40.749616 19480 raft_consensus.cc:2466] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91192ab5da56454c9a8ded8e5d24b782 in term 0.
I20250904 01:58:40.749720 19633 raft_consensus.cc:2466] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91192ab5da56454c9a8ded8e5d24b782 in term 0.
I20250904 01:58:40.750944 19724 leader_election.cc:304] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68d1e8697f2a436880371e195ef692e4, 91192ab5da56454c9a8ded8e5d24b782; no voters:
I20250904 01:58:40.751622 19879 raft_consensus.cc:2802] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:58:40.751973 19879 raft_consensus.cc:491] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:40.752336 19879 raft_consensus.cc:3058] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:40.757304 19879 raft_consensus.cc:513] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.758620 19879 leader_election.cc:290] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [CANDIDATE]: Term 1 election: Requested vote from peers 68d1e8697f2a436880371e195ef692e4 (127.16.112.129:38241), 99367d2d849740cfb2c8275cf5c932ba (127.16.112.130:35497)
I20250904 01:58:40.759337 19480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b365aecdaa6b403abaccba90c6b322a7" candidate_uuid: "91192ab5da56454c9a8ded8e5d24b782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d1e8697f2a436880371e195ef692e4"
I20250904 01:58:40.759462 19633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b365aecdaa6b403abaccba90c6b322a7" candidate_uuid: "91192ab5da56454c9a8ded8e5d24b782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99367d2d849740cfb2c8275cf5c932ba"
I20250904 01:58:40.759703 19480 raft_consensus.cc:3058] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:40.759898 19633 raft_consensus.cc:3058] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:58:40.763800 19480 raft_consensus.cc:2466] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91192ab5da56454c9a8ded8e5d24b782 in term 1.
I20250904 01:58:40.763940 19633 raft_consensus.cc:2466] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91192ab5da56454c9a8ded8e5d24b782 in term 1.
I20250904 01:58:40.764557 19724 leader_election.cc:304] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68d1e8697f2a436880371e195ef692e4, 91192ab5da56454c9a8ded8e5d24b782; no voters:
I20250904 01:58:40.765199 19879 raft_consensus.cc:2802] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 1 FOLLOWER]: Leader election won for term 1
W20250904 01:58:40.765604 19841 tablet.cc:2378] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:58:40.766789 19879 raft_consensus.cc:695] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [term 1 LEADER]: Becoming Leader. State: Replica: 91192ab5da56454c9a8ded8e5d24b782, State: Running, Role: LEADER
I20250904 01:58:40.767840 19879 consensus_queue.cc:237] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } }
I20250904 01:58:40.779731 19122 catalog_manager.cc:5582] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91192ab5da56454c9a8ded8e5d24b782 (127.16.112.131). New cstate: current_term: 1 leader_uuid: "91192ab5da56454c9a8ded8e5d24b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91192ab5da56454c9a8ded8e5d24b782" member_type: VOTER last_known_addr { host: "127.16.112.131" port: 35845 } health_report { overall_health: HEALTHY } } }
I20250904 01:58:41.243124 19879 consensus_queue.cc:1035] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99367d2d849740cfb2c8275cf5c932ba" member_type: VOTER last_known_addr { host: "127.16.112.130" port: 35497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:58:41.258708 19882 consensus_queue.cc:1035] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d1e8697f2a436880371e195ef692e4" member_type: VOTER last_known_addr { host: "127.16.112.129" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:58:41.291702 19891 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:41.292181 19891 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.293669 19892 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:41.294128 19892 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.294420 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:41.295581 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:41.295231 19891 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:41.296113 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 3.
I20250904 01:58:41.296530 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:41.297170 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41; no voters: a616982f94594b49be0986767f8616c1
I20250904 01:58:41.297551 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41" is_pre_election: true
I20250904 01:58:41.297994 19892 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:41.298126 19206 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dae92e829f24e178cf3cb3479bf09b7 in term 3.
I20250904 01:58:41.298547 19891 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 01:58:41.298887 19891 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:41.299263 19891 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:58:41.299134 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41; no voters: a616982f94594b49be0986767f8616c1
I20250904 01:58:41.299871 19892 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 01:58:41.300223 19892 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a616982f94594b49be0986767f8616c1)
I20250904 01:58:41.300588 19892 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:58:41.304389 19891 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.306435 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:41.306520 19891 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:41.306716 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7"
I20250904 01:58:41.309167 19892 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.310166 19285 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c948e45d03324d00b72a04b2848e8a41 in current term 4: Already voted for candidate 9dae92e829f24e178cf3cb3479bf09b7 in this term.
I20250904 01:58:41.310959 19892 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:41.311637 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:41.313855 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c948e45d03324d00b72a04b2848e8a41; no voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1
I20250904 01:58:41.314316 19206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "c948e45d03324d00b72a04b2848e8a41"
I20250904 01:58:41.314554 19891 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250904 01:58:41.315274 19206 raft_consensus.cc:2391] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dae92e829f24e178cf3cb3479bf09b7 in current term 4: Already voted for candidate c948e45d03324d00b72a04b2848e8a41 in this term.
I20250904 01:58:41.316040 19262 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7; no voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:41.316746 19892 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250904 01:58:41.362454 19206 raft_consensus.cc:1238] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Rejecting Update request from peer a616982f94594b49be0986767f8616c1 for earlier term 3. Current term is 4. Ops: []
I20250904 01:58:41.363543 19893 consensus_queue.cc:1046] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 }, Status: INVALID_TERM, Last received: 3.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250904 01:58:41.364593 19894 raft_consensus.cc:3053] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250904 01:58:41.364810 19894 raft_consensus.cc:738] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a616982f94594b49be0986767f8616c1, State: Running, Role: LEADER
I20250904 01:58:41.365392 19894 consensus_queue.cc:260] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.366284 19894 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:58:41.864225 19897 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:58:41.864599 19897 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.866011 19897 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:41.866917 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:41.867102 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:41.867493 19131 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 4.
I20250904 01:58:41.867602 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 4.
I20250904 01:58:41.868389 19182 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a616982f94594b49be0986767f8616c1, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:41.869112 19897 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250904 01:58:41.869413 19897 raft_consensus.cc:491] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:58:41.869697 19897 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 4 FOLLOWER]: Advancing to term 5
I20250904 01:58:41.874373 19897 raft_consensus.cc:513] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.875926 19897 leader_election.cc:290] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 5 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:41.876487 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:41.876963 19131 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 4 FOLLOWER]: Advancing to term 5
I20250904 01:58:41.876863 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c948e45d03324d00b72a04b2848e8a41" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7"
I20250904 01:58:41.877363 19285 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 4 FOLLOWER]: Advancing to term 5
I20250904 01:58:41.882344 19285 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 5.
I20250904 01:58:41.882598 19131 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c948e45d03324d00b72a04b2848e8a41 in term 5.
I20250904 01:58:41.883169 19180 leader_election.cc:304] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41; no voters:
I20250904 01:58:41.884069 19897 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 5 FOLLOWER]: Leader election won for term 5
I20250904 01:58:41.885182 19897 raft_consensus.cc:695] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [term 5 LEADER]: Becoming Leader. State: Replica: c948e45d03324d00b72a04b2848e8a41, State: Running, Role: LEADER
I20250904 01:58:41.885818 19897 consensus_queue.cc:237] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:41.888834 19898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.889273 19898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:41.890362 19900 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:41.897722 19900 catalog_manager.cc:671] Loaded metadata for table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]
I20250904 01:58:41.901877 19900 tablet_loader.cc:96] loaded metadata for tablet b365aecdaa6b403abaccba90c6b322a7 (table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38])
I20250904 01:58:41.902817 19900 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:41.905346 19900 catalog_manager.cc:1261] Loaded cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:41.905534 19900 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:41.908483 19900 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:41.911502 19900 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41: Loaded TSK: 0
I20250904 01:58:41.912714 19900 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:41.915958 19900 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:41.918408 19900 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:58:41.924715 19226 generic_service.cc:175] {username='slave'} at 127.0.0.1:56580 changed flags via RPC: catalog_manager_fail_ts_rpcs from 'false' to 'true'
W20250904 01:58:41.929432 19848 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.112.190:40807)
I20250904 01:58:41.932976 19195 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54144:
table { table_name: "default.test_kudu_1374" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "new_col" type: INT32 is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20250904 01:58:41.944559 19131 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Refusing update from remote peer c948e45d03324d00b72a04b2848e8a41: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
I20250904 01:58:41.944573 19285 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Refusing update from remote peer c948e45d03324d00b72a04b2848e8a41: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
I20250904 01:58:41.945840 19897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250904 01:58:41.946338 19898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250904 01:58:41.954073 19902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.954809 19902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:41.954666 19903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c948e45d03324d00b72a04b2848e8a41. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.955339 19903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:41.959614 19898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.960296 19898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:41.959934 19908 sys_catalog.cc:455] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.961148 19908 sys_catalog.cc:458] T 00000000000000000000000000000000 P c948e45d03324d00b72a04b2848e8a41 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:41.961315 19902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.962272 19902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:41.965063 19903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c948e45d03324d00b72a04b2848e8a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:41.965601 19903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:42.190485 19195 catalog_manager.cc:3973] altered HMS schema for table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]
W20250904 01:58:42.191216 19195 catalog_manager.cc:5842] Failed to send alter table request: Runtime error: Async RPCs configured to fail
I20250904 01:58:42.193109 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19158
W20250904 01:58:42.224614 19727 connection.cc:537] client connection to 127.16.112.189:38857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:58:42.225512 19837 heartbeater.cc:646] Failed to heartbeat to 127.16.112.189:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 01:58:42.231561 19849 master_proxy_rpc.cc:203] Re-attempting IsAlterTableInProgres request to leader Master (127.16.112.189:38857)
W20250904 01:58:42.279495 19679 heartbeater.cc:646] Failed to heartbeat to 127.16.112.189:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
W20250904 01:58:42.295149 19528 heartbeater.cc:646] Failed to heartbeat to 127.16.112.189:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:42.478181 19915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:42.478570 19915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:42.479065 19916 raft_consensus.cc:491] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:42.479446 19916 raft_consensus.cc:513] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:42.480453 19915 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:42.481130 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1" is_pre_election: true
I20250904 01:58:42.481681 19131 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dae92e829f24e178cf3cb3479bf09b7 in term 5.
I20250904 01:58:42.482745 19916 leader_election.cc:290] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857), 9dae92e829f24e178cf3cb3479bf09b7 (127.16.112.188:38591)
I20250904 01:58:42.482690 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1; no voters:
W20250904 01:58:42.483130 19262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:42.483494 19915 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250904 01:58:42.483583 19285 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a616982f94594b49be0986767f8616c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: false dest_uuid: "9dae92e829f24e178cf3cb3479bf09b7" is_pre_election: true
I20250904 01:58:42.483801 19915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c948e45d03324d00b72a04b2848e8a41)
I20250904 01:58:42.484303 19915 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 5 FOLLOWER]: Advancing to term 6
W20250904 01:58:42.484877 19109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20250904 01:58:42.486521 19262 leader_election.cc:336] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857): Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
W20250904 01:58:42.487782 19109 leader_election.cc:336] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857): Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:42.490155 19915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:42.490955 19285 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a616982f94594b49be0986767f8616c1 in current term 6: Already voted for candidate 9dae92e829f24e178cf3cb3479bf09b7 in this term.
I20250904 01:58:42.491544 19915 leader_election.cc:290] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 election: Requested vote from peers a616982f94594b49be0986767f8616c1 (127.16.112.190:40807), c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857)
I20250904 01:58:42.492202 19131 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dae92e829f24e178cf3cb3479bf09b7" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: false dest_uuid: "a616982f94594b49be0986767f8616c1"
I20250904 01:58:42.492229 19106 leader_election.cc:304] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a616982f94594b49be0986767f8616c1; no voters: 9dae92e829f24e178cf3cb3479bf09b7, c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:42.492755 19131 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 5 FOLLOWER]: Advancing to term 6
W20250904 01:58:42.494334 19262 leader_election.cc:336] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857): Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111)
I20250904 01:58:42.497063 19131 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dae92e829f24e178cf3cb3479bf09b7 in term 6.
I20250904 01:58:42.497416 19916 raft_consensus.cc:2747] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 01:58:42.497841 19261 leader_election.cc:304] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dae92e829f24e178cf3cb3479bf09b7, a616982f94594b49be0986767f8616c1; no voters: c948e45d03324d00b72a04b2848e8a41
I20250904 01:58:42.499112 19915 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 6 FOLLOWER]: Leader election won for term 6
I20250904 01:58:42.500954 19915 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [term 6 LEADER]: Becoming Leader. State: Replica: 9dae92e829f24e178cf3cb3479bf09b7, State: Running, Role: LEADER
I20250904 01:58:42.501869 19915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.11, Last appended by leader: 11, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } }
I20250904 01:58:42.504638 19921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dae92e829f24e178cf3cb3479bf09b7. Latest consensus state: current_term: 6 leader_uuid: "9dae92e829f24e178cf3cb3479bf09b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:42.505142 19921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:42.507035 19922 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:58:42.513306 19922 catalog_manager.cc:671] Loaded metadata for table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]
I20250904 01:58:42.517731 19922 tablet_loader.cc:96] loaded metadata for tablet b365aecdaa6b403abaccba90c6b322a7 (table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38])
I20250904 01:58:42.518664 19922 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:58:42.521246 19922 catalog_manager.cc:1261] Loaded cluster ID: da2c7c5f8da24e5ab395fad8965adc61
I20250904 01:58:42.521414 19922 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:58:42.524330 19922 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:58:42.527509 19922 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: Loaded TSK: 0
I20250904 01:58:42.528956 19922 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:58:42.531673 19922 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:58:42.534286 19922 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 1
W20250904 01:58:42.883867 19262 consensus_peers.cc:489] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 -> Peer c948e45d03324d00b72a04b2848e8a41 (127.16.112.189:38857): Couldn't send request to peer c948e45d03324d00b72a04b2848e8a41. Status: Network error: Client connection negotiation failed: client connection to 127.16.112.189:38857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:58:43.106752 19131 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [term 6 FOLLOWER]: Refusing update from remote peer 9dae92e829f24e178cf3cb3479bf09b7: Log matching property violated. Preceding OpId in replica: term: 5 index: 11. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20250904 01:58:43.108158 19921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250904 01:58:43.111796 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dae92e829f24e178cf3cb3479bf09b7. Latest consensus state: current_term: 6 leader_uuid: "9dae92e829f24e178cf3cb3479bf09b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:43.112557 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:43.115880 19928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "9dae92e829f24e178cf3cb3479bf09b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:43.116722 19928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7 [sys.catalog]: This master's current role is: LEADER
I20250904 01:58:43.117444 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "9dae92e829f24e178cf3cb3479bf09b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a616982f94594b49be0986767f8616c1" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 40807 } } peers { permanent_uuid: "c948e45d03324d00b72a04b2848e8a41" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 38857 } } peers { permanent_uuid: "9dae92e829f24e178cf3cb3479bf09b7" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 38591 } } }
I20250904 01:58:43.118245 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P a616982f94594b49be0986767f8616c1 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:58:43.212383 19836 heartbeater.cc:499] Master 127.16.112.188:38591 was elected leader, sending a full tablet report...
I20250904 01:58:43.217259 19274 catalog_manager.cc:5674] TS 91192ab5da56454c9a8ded8e5d24b782 (127.16.112.131:35845) does not have the latest schema for tablet b365aecdaa6b403abaccba90c6b322a7 (table default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38]). Expected version 1 got 0
I20250904 01:58:43.246860 19943 tablet.cc:1722] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
13:new_col INT32 NULLABLE,
PRIMARY KEY (key)
) version 1
I20250904 01:58:43.249749 19943 tablet.cc:1620] T b365aecdaa6b403abaccba90c6b322a7 P 99367d2d849740cfb2c8275cf5c932ba: MemRowSet was empty: no flush needed.
I20250904 01:58:43.249302 19944 tablet.cc:1722] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
13:new_col INT32 NULLABLE,
PRIMARY KEY (key)
) version 1
I20250904 01:58:43.250697 19942 tablet.cc:1722] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NOT NULL,
13:new_col INT32 NULLABLE,
PRIMARY KEY (key)
) version 1
I20250904 01:58:43.252048 19944 tablet.cc:1620] T b365aecdaa6b403abaccba90c6b322a7 P 68d1e8697f2a436880371e195ef692e4: MemRowSet was empty: no flush needed.
I20250904 01:58:43.252843 19942 tablet.cc:1620] T b365aecdaa6b403abaccba90c6b322a7 P 91192ab5da56454c9a8ded8e5d24b782: MemRowSet was empty: no flush needed.
I20250904 01:58:43.257305 19678 heartbeater.cc:499] Master 127.16.112.188:38591 was elected leader, sending a full tablet report...
I20250904 01:58:43.266557 19527 heartbeater.cc:499] Master 127.16.112.188:38591 was elected leader, sending a full tablet report...
I20250904 01:58:43.281023 19274 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9dae92e829f24e178cf3cb3479bf09b7: default.test_kudu_1374 [id=b19e6651a2e2493e8e29e2236aa35d38] alter complete (version 1)
I20250904 01:58:43.606829 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19372
I20250904 01:58:43.625233 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19537
I20250904 01:58:43.651329 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19692
I20250904 01:58:43.678114 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19089
I20250904 01:58:43.706967 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 19232
Shutting down hive metastore.
2025-09-04T01:58:44Z chronyd exiting
[ OK ] HmsConfigurations/MasterFailoverTest.TestKUDU1374/1 (33346 ms)
[ RUN ] HmsConfigurations/MasterFailoverTest.TestMasterPermanentFailure/1
2025-09-04T01:58:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T01:58:44Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/security.properties
2025-09-04 01:58:50: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task_fQbLu/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250904 01:58:57.994390 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:37921
--webserver_interface=127.16.112.190
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--hive_metastore_uris=thrift://127.0.0.1:42699
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
W20250904 01:58:58.282780 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:58:58.283362 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:58:58.283811 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:58:58.313758 20189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:58:58.314092 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:58:58.314342 20189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:58:58.314579 20189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:58:58.348623 20189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42699
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.112.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:58:58.349982 20189 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:58:58.351543 20189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:58:58.362488 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:58.363026 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:58.367408 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:58:59.570219 20197 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1202 milliseconds
I20250904 01:58:59.570369 20189 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:58:59.571656 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:58:59.574100 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:58:59.575430 20189 hybrid_clock.cc:648] HybridClock initialized: now 1756951139575392 us; error 56 us; skew 500 ppm
I20250904 01:58:59.576213 20189 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:58:59.582126 20189 webserver.cc:480] Webserver started at http://127.16.112.190:43207/ using document root <none> and password file <none>
I20250904 01:58:59.583074 20189 fs_manager.cc:362] Metadata directory not provided
I20250904 01:58:59.583271 20189 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:58:59.583717 20189 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:58:59.588112 20189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/instance:
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:58:59 on dist-test-slave-pl7p"
I20250904 01:58:59.589221 20189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance:
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:58:59 on dist-test-slave-pl7p"
I20250904 01:58:59.596310 20189 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250904 01:58:59.601827 20205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:58:59.602819 20189 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 01:58:59.603153 20189 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:58:59 on dist-test-slave-pl7p"
I20250904 01:58:59.603480 20189 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:58:59.655685 20189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:58:59.657251 20189 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:58:59.657727 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:58:59.726975 20189 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.190:37921
I20250904 01:58:59.727032 20256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.190:37921 every 8 connection(s)
I20250904 01:58:59.729725 20189 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
I20250904 01:58:59.734771 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:58:59.739089 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20189
I20250904 01:58:59.739518 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance
I20250904 01:58:59.741811 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:33413
--webserver_interface=127.16.112.189
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--hive_metastore_uris=thrift://127.0.0.1:42699
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:58:59.745844 20257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:58:59.780545 20257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:58:59.782928 20208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
W20250904 01:58:59.786862 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
I20250904 01:58:59.836338 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 1
W20250904 01:58:59.840293 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
I20250904 01:58:59.908730 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 2
W20250904 01:58:59.912662 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
I20250904 01:59:00.004031 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 3
W20250904 01:59:00.007720 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
W20250904 01:59:00.040591 20258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:00.041203 20258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:00.041692 20258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:00.071112 20258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:59:00.071434 20258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:00.071687 20258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:59:00.071919 20258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:59:00.105913 20258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42699
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.189:33413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.112.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:00.107199 20258 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:00.108714 20258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:00.118880 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:00.151275 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 4
W20250904 01:59:00.157315 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
I20250904 01:59:00.456918 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 5
W20250904 01:59:00.462973 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
I20250904 01:59:01.010504 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 6
W20250904 01:59:01.020553 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.189:33413: Network error: Client connection negotiation failed: client connection to 127.16.112.189:33413: connect: Connection refused (error 111)
W20250904 01:59:00.119407 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:01.278787 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:01.280949 20269 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1157 milliseconds
I20250904 01:59:01.281080 20258 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:59:01.282338 20258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:01.284694 20258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:01.286020 20258 hybrid_clock.cc:648] HybridClock initialized: now 1756951141285987 us; error 35 us; skew 500 ppm
I20250904 01:59:01.286753 20258 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:01.293056 20258 webserver.cc:480] Webserver started at http://127.16.112.189:43733/ using document root <none> and password file <none>
I20250904 01:59:01.293992 20258 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:01.294200 20258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:01.294613 20258 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:59:01.298906 20258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data/instance:
uuid: "036c187ff08844b1b3014de468c222a9"
format_stamp: "Formatted at 2025-09-04 01:59:01 on dist-test-slave-pl7p"
I20250904 01:59:01.299942 20258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance:
uuid: "036c187ff08844b1b3014de468c222a9"
format_stamp: "Formatted at 2025-09-04 01:59:01 on dist-test-slave-pl7p"
I20250904 01:59:01.306811 20258 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250904 01:59:01.312083 20279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:01.312971 20258 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250904 01:59:01.313279 20258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal
uuid: "036c187ff08844b1b3014de468c222a9"
format_stamp: "Formatted at 2025-09-04 01:59:01 on dist-test-slave-pl7p"
I20250904 01:59:01.313601 20258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:01.367445 20258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:01.368896 20258 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:01.369324 20258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:01.437399 20258 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.189:33413
I20250904 01:59:01.437443 20330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.189:33413 every 8 connection(s)
I20250904 01:59:01.440042 20258 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/data/info.pb
I20250904 01:59:01.444722 20331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:59:01.445065 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20258
I20250904 01:59:01.445544 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-1/wal/instance
I20250904 01:59:01.448803 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:36653
--webserver_interface=127.16.112.188
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--hive_metastore_uris=thrift://127.0.0.1:42699
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
I20250904 01:59:01.459091 20331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:59:01.479133 20331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:59:01.492625 20331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:59:01.494920 20281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
W20250904 01:59:01.498687 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:01.548091 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 1
W20250904 01:59:01.551595 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:01.619982 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 2
W20250904 01:59:01.623551 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:01.714885 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 3
W20250904 01:59:01.718623 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
W20250904 01:59:01.745414 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:01.746026 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:01.746523 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:01.776011 20332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:59:01.776268 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:01.776460 20332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:59:01.776711 20332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:59:01.809231 20332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42699
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.188:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.112.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:01.810410 20332 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:01.811908 20332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:01.822484 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:01.861989 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 4
W20250904 01:59:01.869482 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.081202 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } attempt: 7
I20250904 01:59:02.130111 20257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
W20250904 01:59:02.161418 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.169039 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 5
W20250904 01:59:02.181692 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.220070 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 1
W20250904 01:59:02.230545 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.312143 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 2
W20250904 01:59:02.319841 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.405457 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 3
W20250904 01:59:02.413378 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.554028 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 4
W20250904 01:59:02.562292 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.729138 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 6
W20250904 01:59:02.760075 20331 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
I20250904 01:59:02.845854 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 5
W20250904 01:59:02.851387 20257 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.16.112.188:36653: Network error: Client connection negotiation failed: client connection to 127.16.112.188:36653: connect: Connection refused (error 111)
W20250904 01:59:01.823305 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:03.094717 20332 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.272s user 0.397s sys 0.859s
W20250904 01:59:03.095048 20332 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.273s user 0.397s sys 0.859s
W20250904 01:59:03.095760 20344 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1271 milliseconds
W20250904 01:59:03.095932 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:03.095918 20332 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:59:03.097769 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:03.101061 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:03.102566 20332 hybrid_clock.cc:648] HybridClock initialized: now 1756951143102515 us; error 36 us; skew 500 ppm
I20250904 01:59:03.103757 20332 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:03.112732 20332 webserver.cc:480] Webserver started at http://127.16.112.188:46851/ using document root <none> and password file <none>
I20250904 01:59:03.114205 20332 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:03.114504 20332 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:03.115134 20332 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:59:03.122495 20332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/instance:
uuid: "c2218317dfa74e5fa78821de6a65878f"
format_stamp: "Formatted at 2025-09-04 01:59:03 on dist-test-slave-pl7p"
I20250904 01:59:03.124042 20332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance:
uuid: "c2218317dfa74e5fa78821de6a65878f"
format_stamp: "Formatted at 2025-09-04 01:59:03 on dist-test-slave-pl7p"
I20250904 01:59:03.133905 20332 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s
I20250904 01:59:03.141819 20358 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:03.143033 20332 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250904 01:59:03.143448 20332 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
uuid: "c2218317dfa74e5fa78821de6a65878f"
format_stamp: "Formatted at 2025-09-04 01:59:03 on dist-test-slave-pl7p"
I20250904 01:59:03.143905 20332 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:03.271433 20332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:03.272812 20332 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:03.273206 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:03.341197 20332 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.188:36653
I20250904 01:59:03.341265 20409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.188:36653 every 8 connection(s)
I20250904 01:59:03.343662 20332 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/info.pb
I20250904 01:59:03.347327 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20332
I20250904 01:59:03.347819 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal/instance
I20250904 01:59:03.348888 20410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 01:59:03.364718 20410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:59:03.384387 20410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:59:03.393529 20410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
I20250904 01:59:03.403807 20257 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 6
I20250904 01:59:03.409993 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Bootstrap starting.
I20250904 01:59:03.416563 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Neither blocks nor log segments found. Creating new log.
I20250904 01:59:03.418267 20410 log.cc:826] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Log is configured to *not* fsync() on all Append() calls
I20250904 01:59:03.422268 20257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Bootstrap starting.
I20250904 01:59:03.423008 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: No bootstrap required, opened a new log
I20250904 01:59:03.427248 20257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Neither blocks nor log segments found. Creating new log.
I20250904 01:59:03.428929 20257 log.cc:826] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Log is configured to *not* fsync() on all Append() calls
I20250904 01:59:03.434459 20257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: No bootstrap required, opened a new log
I20250904 01:59:03.445722 20410 raft_consensus.cc:357] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.446669 20410 raft_consensus.cc:383] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:59:03.447100 20410 raft_consensus.cc:738] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2218317dfa74e5fa78821de6a65878f, State: Initialized, Role: FOLLOWER
I20250904 01:59:03.448082 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.451610 20419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.452481 20419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:03.452982 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:59:03.461438 20257 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.462476 20257 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:59:03.462800 20257 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3765bd467a4f4ceea91501064e04a61a, State: Initialized, Role: FOLLOWER
I20250904 01:59:03.463778 20257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.467137 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.467877 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:03.469287 20257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: configured and running, proceeding with master startup.
W20250904 01:59:03.514918 20439 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:59:03.515245 20439 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250904 01:59:03.528092 20364 tablet.cc:2378] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:59:03.489042 20332 master_runner.cc:186] Error getting master registration for 127.16.112.190:37921: OK, instance_id { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" instance_seqno: 1756951139708069 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:59:03.538408 20332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:59:03.611737 20451 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250904 01:59:03.612242 20451 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250904 01:59:03.583063 20189 master_runner.cc:186] Error getting master registration for 127.16.112.189:33413: OK, instance_id { permanent_uuid: "036c187ff08844b1b3014de468c222a9" instance_seqno: 1756951141418780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250904 01:59:03.619593 20189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250904 01:59:03.666854 20211 tablet.cc:2378] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 01:59:03.706094 20426 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:59:03.706619 20426 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.710057 20426 leader_election.cc:290] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:03.710345 20385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3765bd467a4f4ceea91501064e04a61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:03.711210 20385 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3765bd467a4f4ceea91501064e04a61a in term 0.
I20250904 01:59:03.712463 20207 leader_election.cc:304] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f; no voters:
I20250904 01:59:03.713543 20426 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 01:59:03.713934 20426 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:59:03.714270 20426 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:59:03.720968 20426 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.722973 20426 leader_election.cc:290] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [CANDIDATE]: Term 1 election: Requested vote from peers 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:03.724200 20385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3765bd467a4f4ceea91501064e04a61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f"
I20250904 01:59:03.724701 20385 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:59:03.733814 20385 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3765bd467a4f4ceea91501064e04a61a in term 1.
I20250904 01:59:03.741845 20207 leader_election.cc:304] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f; no voters:
I20250904 01:59:03.742664 20426 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 01:59:03.762228 20426 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 1 LEADER]: Becoming Leader. State: Replica: 3765bd467a4f4ceea91501064e04a61a, State: Running, Role: LEADER
I20250904 01:59:03.764753 20426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.792567 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3765bd467a4f4ceea91501064e04a61a. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.793462 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:03.799252 20455 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:59:03.806357 20455 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:59:03.820649 20331 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } attempt: 7
I20250904 01:59:03.836150 20385 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Refusing update from remote peer 3765bd467a4f4ceea91501064e04a61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:59:03.838109 20453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:59:03.874246 20331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Bootstrap starting.
I20250904 01:59:03.887991 20331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Neither blocks nor log segments found. Creating new log.
I20250904 01:59:03.890235 20331 log.cc:826] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Log is configured to *not* fsync() on all Append() calls
I20250904 01:59:03.899853 20419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3765bd467a4f4ceea91501064e04a61a. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.900501 20419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:03.917343 20331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: No bootstrap required, opened a new log
I20250904 01:59:03.925432 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.926144 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:03.933593 20455 catalog_manager.cc:1349] Generated new cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:03.933887 20455 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:59:03.958393 20419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.959115 20419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:03.967531 20455 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 01:59:03.968278 20331 raft_consensus.cc:357] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.969122 20331 raft_consensus.cc:383] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 01:59:03.969417 20331 raft_consensus.cc:738] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036c187ff08844b1b3014de468c222a9, State: Initialized, Role: FOLLOWER
I20250904 01:59:03.970311 20331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:03.980878 20455 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:59:03.986546 20465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:03.987306 20465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:03.988703 20331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:59:03.995630 20306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3765bd467a4f4ceea91501064e04a61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:03.996492 20306 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3765bd467a4f4ceea91501064e04a61a in term 0.
I20250904 01:59:03.998214 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3765bd467a4f4ceea91501064e04a61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9"
I20250904 01:59:03.998771 20305 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 0 FOLLOWER]: Advancing to term 1
I20250904 01:59:04.007006 20305 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3765bd467a4f4ceea91501064e04a61a in term 1.
I20250904 01:59:04.008648 20304 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Refusing update from remote peer 3765bd467a4f4ceea91501064e04a61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 01:59:04.015878 20453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 01:59:04.054697 20455 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Generated new TSK 0
I20250904 01:59:04.055826 20455 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:59:04.060024 20455 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:59:04.063035 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:04.063766 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:04.064478 20465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3765bd467a4f4ceea91501064e04a61a. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:04.064509 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3765bd467a4f4ceea91501064e04a61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:04.065192 20465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:04.065423 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:04.067157 20455 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:59:04.078904 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.129:0
--local_ip_for_outbound_sockets=127.16.112.129
--webserver_interface=127.16.112.129
--webserver_port=0
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:59:04.158999 20487 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:04.159359 20487 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: loading cluster ID for follower catalog manager: success
I20250904 01:59:04.167019 20487 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: acquiring CA information for follower catalog manager: success
I20250904 01:59:04.186259 20487 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:59:04.520488 20439 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:04.520843 20439 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: loading cluster ID for follower catalog manager: success
I20250904 01:59:04.527702 20439 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: acquiring CA information for follower catalog manager: success
I20250904 01:59:04.539175 20439 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250904 01:59:04.625928 20486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:04.626575 20486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:04.627521 20486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:04.684088 20486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:04.685623 20486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.129
I20250904 01:59:04.748211 20486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.112.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.129
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:04.749857 20486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:04.751780 20486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:04.791271 20494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:04.982846 20500 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:04.989800 20500 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:04.993002 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:04.995019 20500 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:04.996376 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:04.998683 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:05.004103 20500 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:05.550083 20501 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:05.550799 20501 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:05.553520 20501 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:05.554437 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:05.556326 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:05.558624 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:05.559531 20501 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:05.966866 20502 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:05.967686 20502 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:05.970543 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:05.983080 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:05.984836 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:05.987444 20502 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:05.990080 20502 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:06.540427 20504 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:06.541121 20504 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:06.543908 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:06.546886 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:06.548449 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:06.550993 20504 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:06.551810 20504 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:07.008467 20506 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:07.010375 20506 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:07.014573 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:07.016147 20506 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:07.017199 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:07.020356 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:07.021070 20506 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:59:06.172878 20493 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 20486
W20250904 01:59:07.115512 20497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:07.116027 20496 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2327 milliseconds
W20250904 01:59:07.116264 20486 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.326s user 0.326s sys 0.773s
W20250904 01:59:07.118098 20486 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.328s user 0.326s sys 0.773s
I20250904 01:59:07.118356 20486 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 01:59:04.792308 20495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:07.116977 20507 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:07.119325 20507 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:07.122262 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:07.122499 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:07.123934 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:07.124933 20486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:07.128821 20507 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:07.129321 20507 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:07.134604 20486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:07.138211 20486 hybrid_clock.cc:648] HybridClock initialized: now 1756951147138157 us; error 54 us; skew 500 ppm
I20250904 01:59:07.139284 20486 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:07.147002 20486 webserver.cc:480] Webserver started at http://127.16.112.129:38707/ using document root <none> and password file <none>
I20250904 01:59:07.148247 20486 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:07.148522 20486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:07.149096 20486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:59:07.155328 20486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data/instance:
uuid: "49c95031b549406f9b2cf96a060d90a7"
format_stamp: "Formatted at 2025-09-04 01:59:07 on dist-test-slave-pl7p"
I20250904 01:59:07.156801 20486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance:
uuid: "49c95031b549406f9b2cf96a060d90a7"
format_stamp: "Formatted at 2025-09-04 01:59:07 on dist-test-slave-pl7p"
I20250904 01:59:07.166198 20486 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.006s
I20250904 01:59:07.174028 20513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:07.175338 20486 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250904 01:59:07.175742 20486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
uuid: "49c95031b549406f9b2cf96a060d90a7"
format_stamp: "Formatted at 2025-09-04 01:59:07 on dist-test-slave-pl7p"
I20250904 01:59:07.176172 20486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:07.260141 20486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:07.262104 20486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:07.262639 20486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:07.265964 20486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:59:07.271806 20486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:59:07.272078 20486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:07.272387 20486 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:59:07.272622 20486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:07.522630 20486 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.129:36551
I20250904 01:59:07.526228 20486 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/data/info.pb
I20250904 01:59:07.528601 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20486
I20250904 01:59:07.529064 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-0/wal/instance
I20250904 01:59:07.530372 20626 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:07.530990 20626 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:07.535123 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:07.547184 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:07.549162 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.130:0
--local_ip_for_outbound_sockets=127.16.112.130
--webserver_interface=127.16.112.130
--webserver_port=0
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:59:07.549259 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:07.522750 20625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.129:36551 every 8 connection(s)
I20250904 01:59:07.558185 20626 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:07.559773 20626 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:07.598233 20627 heartbeater.cc:344] Connected to a master server at 127.16.112.188:36653
I20250904 01:59:07.598915 20627 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:07.600441 20627 heartbeater.cc:507] Master 127.16.112.188:36653 requested a full tablet report, sending...
I20250904 01:59:07.604451 20375 ts_manager.cc:194] Registered new tserver with Master: 49c95031b549406f9b2cf96a060d90a7 (127.16.112.129:36551)
I20250904 01:59:07.631357 20628 heartbeater.cc:344] Connected to a master server at 127.16.112.189:33413
I20250904 01:59:07.631742 20628 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:07.632577 20628 heartbeater.cc:507] Master 127.16.112.189:33413 requested a full tablet report, sending...
I20250904 01:59:07.635390 20296 ts_manager.cc:194] Registered new tserver with Master: 49c95031b549406f9b2cf96a060d90a7 (127.16.112.129:36551)
I20250904 01:59:07.647428 20633 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:07.647787 20633 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:07.648670 20633 heartbeater.cc:507] Master 127.16.112.190:37921 requested a full tablet report, sending...
I20250904 01:59:07.651192 20222 ts_manager.cc:194] Registered new tserver with Master: 49c95031b549406f9b2cf96a060d90a7 (127.16.112.129:36551)
I20250904 01:59:07.653784 20222 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.129:60119
W20250904 01:59:07.986184 20632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:07.986662 20632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:07.987284 20632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:08.018689 20632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:08.019654 20632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.130
I20250904 01:59:08.055815 20632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.112.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.130
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:08.057116 20632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:08.058707 20632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:08.071060 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:08.157498 20633 heartbeater.cc:499] Master 127.16.112.190:37921 was elected leader, sending a full tablet report...
I20250904 01:59:08.885782 20650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:08.886703 20650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:08.890623 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:08.895144 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:08.897346 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:08.900645 20650 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:08.900929 20651 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:09.007500 20652 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:09.008328 20652 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:09.012095 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:09.016009 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:09.018098 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:09.023360 20652 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:09.024555 20652 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:59:09.473399 20642 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 20632
W20250904 01:59:09.830040 20642 kernel_stack_watchdog.cc:198] Thread 20632 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 01:59:09.830276 20632 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.759s user 0.677s sys 1.033s
W20250904 01:59:08.071643 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:09.830627 20632 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.759s user 0.677s sys 1.034s
W20250904 01:59:09.832734 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:09.835307 20645 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1759 milliseconds
I20250904 01:59:09.835366 20632 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:59:09.836460 20632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:09.838465 20632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:09.839788 20632 hybrid_clock.cc:648] HybridClock initialized: now 1756951149839752 us; error 33 us; skew 500 ppm
I20250904 01:59:09.840523 20632 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:09.846267 20632 webserver.cc:480] Webserver started at http://127.16.112.130:33819/ using document root <none> and password file <none>
I20250904 01:59:09.847177 20632 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:09.847383 20632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:09.847790 20632 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:59:09.852067 20632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data/instance:
uuid: "d38c47cca76d44c6b7f52e18d380ae16"
format_stamp: "Formatted at 2025-09-04 01:59:09 on dist-test-slave-pl7p"
I20250904 01:59:09.853099 20632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance:
uuid: "d38c47cca76d44c6b7f52e18d380ae16"
format_stamp: "Formatted at 2025-09-04 01:59:09 on dist-test-slave-pl7p"
I20250904 01:59:09.859983 20632 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250904 01:59:09.865206 20658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:09.866334 20632 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250904 01:59:09.866659 20632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
uuid: "d38c47cca76d44c6b7f52e18d380ae16"
format_stamp: "Formatted at 2025-09-04 01:59:09 on dist-test-slave-pl7p"
I20250904 01:59:09.866945 20632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:09.918524 20632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:09.919929 20632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:09.920351 20632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:09.923094 20632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:59:09.927167 20632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:59:09.927371 20632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:09.927629 20632 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:59:09.927842 20632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:09.929271 20665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:09.929737 20665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:09.931130 20665 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:09.931792 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:09.932150 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:09.933553 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:09.934300 20665 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:10.064538 20632 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.130:45285
I20250904 01:59:10.064649 20772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.130:45285 every 8 connection(s)
I20250904 01:59:10.067198 20632 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/data/info.pb
I20250904 01:59:10.068576 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20632
I20250904 01:59:10.068948 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-1/wal/instance
I20250904 01:59:10.080487 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.112.131:0
--local_ip_for_outbound_sockets=127.16.112.131
--webserver_interface=127.16.112.131
--webserver_port=0
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250904 01:59:10.111681 20773 heartbeater.cc:344] Connected to a master server at 127.16.112.188:36653
I20250904 01:59:10.112263 20773 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:10.113637 20773 heartbeater.cc:507] Master 127.16.112.188:36653 requested a full tablet report, sending...
I20250904 01:59:10.116379 20375 ts_manager.cc:194] Registered new tserver with Master: d38c47cca76d44c6b7f52e18d380ae16 (127.16.112.130:45285)
I20250904 01:59:10.116937 20774 heartbeater.cc:344] Connected to a master server at 127.16.112.189:33413
I20250904 01:59:10.117257 20774 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:10.118141 20774 heartbeater.cc:507] Master 127.16.112.189:33413 requested a full tablet report, sending...
I20250904 01:59:10.119784 20775 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:10.120092 20775 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:10.120924 20775 heartbeater.cc:507] Master 127.16.112.190:37921 requested a full tablet report, sending...
I20250904 01:59:10.121167 20296 ts_manager.cc:194] Registered new tserver with Master: d38c47cca76d44c6b7f52e18d380ae16 (127.16.112.130:45285)
I20250904 01:59:10.122923 20222 ts_manager.cc:194] Registered new tserver with Master: d38c47cca76d44c6b7f52e18d380ae16 (127.16.112.130:45285)
I20250904 01:59:10.124181 20222 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.130:58303
W20250904 01:59:10.413758 20781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:10.414227 20781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:10.414803 20781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:10.455260 20781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:10.456796 20781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.112.131
I20250904 01:59:10.501905 20781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.112.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.112.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.112.131
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:10.503123 20781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:10.504604 20781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:10.515812 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:10.628939 20775 heartbeater.cc:499] Master 127.16.112.190:37921 was elected leader, sending a full tablet report...
I20250904 01:59:10.928241 20795 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:10.929275 20795 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:10.933403 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:10.939175 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:10.941797 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:10.946941 20795 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:10.948282 20795 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:11.375341 20797 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:11.384743 20797 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:11.388388 20797 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:11.388391 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:11.388639 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:11.390621 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:11.392231 20797 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:11.836476 20799 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:11.837989 20799 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:11.842360 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:11.846150 20799 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:11.846284 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:11.848434 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:11.850025 20799 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 01:59:10.516498 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:12.085244 20781 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.569s user 0.513s sys 0.996s
W20250904 01:59:11.920902 20788 debug-util.cc:398] Leaking SignalData structure 0x7b0800037ce0 after lost signal to thread 20781
W20250904 01:59:12.085759 20781 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.570s user 0.514s sys 0.996s
W20250904 01:59:12.087939 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:12.092839 20791 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1576 milliseconds
I20250904 01:59:12.092859 20781 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:59:12.094065 20781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:12.095965 20781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:12.097297 20781 hybrid_clock.cc:648] HybridClock initialized: now 1756951152097255 us; error 46 us; skew 500 ppm
I20250904 01:59:12.098115 20781 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:12.103960 20781 webserver.cc:480] Webserver started at http://127.16.112.131:38239/ using document root <none> and password file <none>
I20250904 01:59:12.104867 20781 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:12.105078 20781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:12.105480 20781 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 01:59:12.109737 20781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data/instance:
uuid: "c6a9423f201e49aaa10da3614d7169a2"
format_stamp: "Formatted at 2025-09-04 01:59:12 on dist-test-slave-pl7p"
I20250904 01:59:12.110787 20781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance:
uuid: "c6a9423f201e49aaa10da3614d7169a2"
format_stamp: "Formatted at 2025-09-04 01:59:12 on dist-test-slave-pl7p"
I20250904 01:59:12.117435 20781 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20250904 01:59:12.122759 20804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:12.123736 20781 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250904 01:59:12.124027 20781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
uuid: "c6a9423f201e49aaa10da3614d7169a2"
format_stamp: "Formatted at 2025-09-04 01:59:12 on dist-test-slave-pl7p"
I20250904 01:59:12.124349 20781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:12.176406 20781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:12.178119 20781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:12.178512 20781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:12.180922 20781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 01:59:12.184896 20781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 01:59:12.185081 20781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:12.185281 20781 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 01:59:12.185436 20781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:12.322443 20781 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.131:40447
I20250904 01:59:12.322554 20916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.131:40447 every 8 connection(s)
I20250904 01:59:12.324901 20781 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/data/info.pb
I20250904 01:59:12.333565 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 20781
I20250904 01:59:12.334158 16834 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/ts-2/wal/instance
I20250904 01:59:12.373199 20918 heartbeater.cc:344] Connected to a master server at 127.16.112.189:33413
I20250904 01:59:12.373669 20918 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:12.374914 20918 heartbeater.cc:507] Master 127.16.112.189:33413 requested a full tablet report, sending...
I20250904 01:59:12.381911 20917 heartbeater.cc:344] Connected to a master server at 127.16.112.188:36653
I20250904 01:59:12.382283 20917 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:12.383193 20917 heartbeater.cc:507] Master 127.16.112.188:36653 requested a full tablet report, sending...
I20250904 01:59:12.377782 20927 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:12.384542 20927 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:12.386237 20375 ts_manager.cc:194] Registered new tserver with Master: c6a9423f201e49aaa10da3614d7169a2 (127.16.112.131:40447)
I20250904 01:59:12.387826 20927 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:12.388908 20296 ts_manager.cc:194] Registered new tserver with Master: c6a9423f201e49aaa10da3614d7169a2 (127.16.112.131:40447)
I20250904 01:59:12.389127 20232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:12.390792 20919 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:12.391096 20919 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:12.391266 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:12.391970 20919 heartbeater.cc:507] Master 127.16.112.190:37921 requested a full tablet report, sending...
I20250904 01:59:12.392815 20281 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:12.394268 20927 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 01:59:12.394941 20221 ts_manager.cc:194] Registered new tserver with Master: c6a9423f201e49aaa10da3614d7169a2 (127.16.112.131:40447)
I20250904 01:59:12.396831 20221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.112.131:55321
I20250904 01:59:12.412072 16834 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 01:59:12.446458 16834 test_util.cc:276] Using random seed: 290737119
I20250904 01:59:12.446847 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20189
W20250904 01:59:12.660742 20775 heartbeater.cc:646] Failed to heartbeat to 127.16.112.190:37921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
W20250904 01:59:12.701773 20633 heartbeater.cc:646] Failed to heartbeat to 127.16.112.190:37921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:12.809747 20935 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:12.810118 20935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250904 01:59:12.813439 20935 dir_manager.cc:729] IO error: Could not lock instance file. Make sure that Kudu is not already running and you are not trying to run Kudu with a different user than before: lock /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data/data/block_manager_instance: Resource temporarily unavailable (error 11)
W20250904 01:59:12.813650 20935 dir_manager.cc:730] Proceeding without lock
I20250904 01:59:12.818082 20935 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.005s
I20250904 01:59:12.818388 20935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-2/wal
uuid: "c2218317dfa74e5fa78821de6a65878f"
format_stamp: "Formatted at 2025-09-04 01:59:03 on dist-test-slave-pl7p"
I20250904 01:59:12.879387 20944 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:12.879803 20944 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:12.881189 20944 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
W20250904 01:59:12.883070 20361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:12.883086 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:12.883582 20305 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2218317dfa74e5fa78821de6a65878f in term 1.
I20250904 01:59:12.885296 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036c187ff08844b1b3014de468c222a9, c2218317dfa74e5fa78821de6a65878f; no voters:
I20250904 01:59:12.886021 20944 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 01:59:12.886241 20944 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3765bd467a4f4ceea91501064e04a61a)
I20250904 01:59:12.886504 20944 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 1 FOLLOWER]: Advancing to term 2
W20250904 01:59:12.887264 20361 leader_election.cc:336] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:12.892763 20944 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:12.894589 20944 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 election: Requested vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:12.897176 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9"
I20250904 01:59:12.897730 20305 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 1 FOLLOWER]: Advancing to term 2
W20250904 01:59:12.898666 20361 leader_election.cc:336] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:12.899984 20919 heartbeater.cc:499] Master 127.16.112.190:37921 was elected leader, sending a full tablet report...
W20250904 01:59:12.903676 20919 heartbeater.cc:646] Failed to heartbeat to 127.16.112.190:37921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:12.904268 20305 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2218317dfa74e5fa78821de6a65878f in term 2.
I20250904 01:59:12.905012 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 036c187ff08844b1b3014de468c222a9, c2218317dfa74e5fa78821de6a65878f; no voters: 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:12.905613 20944 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 FOLLOWER]: Leader election won for term 2
I20250904 01:59:12.907123 20944 raft_consensus.cc:695] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 LEADER]: Becoming Leader. State: Replica: c2218317dfa74e5fa78821de6a65878f, State: Running, Role: LEADER
I20250904 01:59:12.907861 20944 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:12.911967 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2218317dfa74e5fa78821de6a65878f. Latest consensus state: current_term: 2 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:12.912760 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:12.914634 20950 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:59:12.920011 20950 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:59:12.922760 20950 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:12.922950 20950 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:59:12.925904 20950 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:59:12.928467 20950 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Loaded TSK: 0
I20250904 01:59:12.929388 20950 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:59:12.931751 20950 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:59:12.934135 20950 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
W20250904 01:59:13.408607 20361 consensus_peers.cc:489] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f -> Peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Couldn't send request to peer 3765bd467a4f4ceea91501064e04a61a. Status: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 01:59:13.476565 20305 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Refusing update from remote peer c2218317dfa74e5fa78821de6a65878f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250904 01:59:13.477746 20948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250904 01:59:13.482112 20956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2218317dfa74e5fa78821de6a65878f. Latest consensus state: current_term: 2 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:13.482930 20956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:13.486554 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:13.487293 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:13.487237 20956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:13.488008 20956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:13.645622 20773 heartbeater.cc:499] Master 127.16.112.188:36653 was elected leader, sending a full tablet report...
I20250904 01:59:13.666298 20627 heartbeater.cc:499] Master 127.16.112.188:36653 was elected leader, sending a full tablet report...
I20250904 01:59:13.838713 16834 master_failover-itest.cc:413] tablet: 00000000000000000000000000000000, peers: 3765bd467a4f4ceea91501064e04a61a 036c187ff08844b1b3014de468c222a9 c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:13.900945 20917 heartbeater.cc:499] Master 127.16.112.188:36653 was elected leader, sending a full tablet report...
I20250904 01:59:14.178678 20965 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:14.179049 20965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:14.187795 20965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/instance:
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:59:14 on dist-test-slave-pl7p"
I20250904 01:59:14.192133 20965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal/instance:
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:59:14 on dist-test-slave-pl7p"
I20250904 01:59:14.205094 20965 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.004s sys 0.004s
I20250904 01:59:14.471568 20979 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:14.471974 20979 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:14.473315 20979 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:14.474257 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
W20250904 01:59:14.475437 20282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250904 01:59:14.478443 20282 leader_election.cc:336] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:14.478789 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:14.479288 20979 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250904 01:59:15.518121 21000 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:15.518627 21000 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:15.520586 21000 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:15.522245 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
W20250904 01:59:15.535218 20282 leader_election.cc:336] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:15.535701 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:15.536406 21000 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250904 01:59:15.578764 20994 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:15.579151 20994 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:15.588426 20994 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s
I20250904 01:59:15.592381 21005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:15.593114 20994 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:15.593780 20994 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250904 01:59:15.593964 20994 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:59:14 on dist-test-slave-pl7p"
I20250904 01:59:16.089815 21018 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:16.090679 21018 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:16.093972 21018 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:16.094775 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
W20250904 01:59:16.103061 20282 leader_election.cc:336] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:16.103717 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:16.104775 21018 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250904 01:59:16.199558 20361 consensus_peers.cc:489] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f -> Peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Couldn't send request to peer 3765bd467a4f4ceea91501064e04a61a. Status: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250904 01:59:16.862260 20994 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.227s user 0.428s sys 0.799s
W20250904 01:59:16.862520 20994 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.227s user 0.428s sys 0.799s
W20250904 01:59:16.864544 21008 tool_action_local_replica.cc:348] Start to copy tablet 00000000000000000000000000000000
I20250904 01:59:16.867488 21008 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.112.188:36653
I20250904 01:59:16.888451 20395 tablet_copy_service.cc:140] P c2218317dfa74e5fa78821de6a65878f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3765bd467a4f4ceea91501064e04a61a ({username='slave'} at 127.0.0.1:36452)
I20250904 01:59:16.888877 20395 tablet_copy_service.cc:161] P c2218317dfa74e5fa78821de6a65878f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3765bd467a4f4ceea91501064e04a61a at {username='slave'} at 127.0.0.1:36452: session id = 3765bd467a4f4ceea91501064e04a61a-00000000000000000000000000000000
I20250904 01:59:16.894161 20395 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 01:59:16.899883 21008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250904 01:59:16.930240 21008 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: tablet copy: Starting download of 0 data blocks...
I20250904 01:59:16.932893 21008 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: tablet copy: Starting download of 1 WAL segments...
I20250904 01:59:16.938118 21008 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 01:59:16.950028 21008 tool_action_local_replica.cc:378] Tablet 00000000000000000000000000000000 copy succeed.
I20250904 01:59:16.950331 21008 tool_action_local_replica.cc:386] 1/1 tablets, 1952 bytes copied, include 0 failed tablets.
I20250904 01:59:16.951769 20395 tablet_copy_service.cc:342] P c2218317dfa74e5fa78821de6a65878f: Request end of tablet copy session 3765bd467a4f4ceea91501064e04a61a-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:36452
I20250904 01:59:16.952108 20395 tablet_copy_service.cc:434] P c2218317dfa74e5fa78821de6a65878f: ending tablet copy session 3765bd467a4f4ceea91501064e04a61a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3765bd467a4f4ceea91501064e04a61a
I20250904 01:59:17.526330 21038 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:17.526693 21038 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:17.528172 21038 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:17.529158 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
W20250904 01:59:17.532277 20282 leader_election.cc:336] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:17.532570 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:17.533068 21038 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250904 01:59:17.900184 20361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250904 01:59:17.981685 16834 external_mini_cluster.cc:1366] Running /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
/tmp/dist-test-task_fQbLu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:37921
--webserver_interface=127.16.112.190
--webserver_port=0
--builtin_ntp_servers=127.16.112.148:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--hive_metastore_uris=thrift://127.0.0.1:42699
--leader_failure_max_missed_heartbeat_periods=1.0 with env {}
W20250904 01:59:18.277356 21047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 01:59:18.277949 21047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 01:59:18.278396 21047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 01:59:18.309041 21047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 01:59:18.309376 21047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 01:59:18.309646 21047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 01:59:18.309878 21047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 01:59:18.344028 21047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.112.148:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42699
--ipki_ca_key_size=768
--master_addresses=127.16.112.190:37921,127.16.112.189:33413,127.16.112.188:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.112.190:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.112.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 01:43:22 UTC on 5fd53c4cbb9d
build id 7889
TSAN enabled
I20250904 01:59:18.345283 21047 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 01:59:18.346922 21047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 01:59:18.358242 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:18.856669 20361 consensus_peers.cc:489] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f -> Peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Couldn't send request to peer 3765bd467a4f4ceea91501064e04a61a. Status: Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250904 01:59:19.015265 21065 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:19.016294 21065 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:19.028231 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:19.031348 21065 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
W20250904 01:59:19.053093 20282 leader_election.cc:336] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921): Network error: Client connection negotiation failed: client connection to 127.16.112.190:37921: connect: Connection refused (error 111)
I20250904 01:59:19.053771 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036c187ff08844b1b3014de468c222a9; no voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:19.055176 21065 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250904 01:59:19.760815 21054 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 21047
W20250904 01:59:19.822367 21047 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.464s user 0.410s sys 0.959s
W20250904 01:59:19.822856 21047 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.465s user 0.411s sys 0.959s
W20250904 01:59:18.360363 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 01:59:19.824028 21057 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1463 milliseconds
W20250904 01:59:19.824487 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 01:59:19.824985 21047 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 01:59:19.827746 21047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 01:59:19.830037 21047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 01:59:19.831409 21047 hybrid_clock.cc:648] HybridClock initialized: now 1756951159831362 us; error 41 us; skew 500 ppm
I20250904 01:59:19.832168 21047 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 01:59:19.838025 21047 webserver.cc:480] Webserver started at http://127.16.112.190:37995/ using document root <none> and password file <none>
I20250904 01:59:19.838913 21047 fs_manager.cc:362] Metadata directory not provided
I20250904 01:59:19.839151 21047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 01:59:19.846670 21047 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250904 01:59:19.850852 21077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 01:59:19.851842 21047 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250904 01:59:19.852150 21047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data,/tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
uuid: "3765bd467a4f4ceea91501064e04a61a"
format_stamp: "Formatted at 2025-09-04 01:59:14 on dist-test-slave-pl7p"
I20250904 01:59:19.854043 21047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 01:59:19.913290 21047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 01:59:19.914714 21047 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 01:59:19.915117 21047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 01:59:19.986629 21047 rpc_server.cc:307] RPC server started. Bound to: 127.16.112.190:37921
I20250904 01:59:19.986702 21128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.112.190:37921 every 8 connection(s)
I20250904 01:59:19.989285 21047 server_base.cc:1179] Dumped server information to /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0/minicluster-data/master-0/data/info.pb
I20250904 01:59:19.996382 16834 external_mini_cluster.cc:1428] Started /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu as pid 21047
I20250904 01:59:20.002018 21129 sys_catalog.cc:263] Verifying existing consensus state
I20250904 01:59:20.012995 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Bootstrap starting.
I20250904 01:59:20.024498 20919 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:20.025000 20919 heartbeater.cc:499] Master 127.16.112.190:37921 was elected leader, sending a full tablet report...
I20250904 01:59:20.029863 21129 log.cc:826] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Log is configured to *not* fsync() on all Append() calls
W20250904 01:59:20.033686 20930 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.112.190:37921)
I20250904 01:59:20.033622 21134 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:20.034132 21134 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.037479 21134 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:20.039175 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f" is_pre_election: true
I20250904 01:59:20.040247 20375 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41628:
name: "default.table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 01:59:20.051112 20375 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 01:59:20.054781 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 01:59:20.055660 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: Bootstrap complete.
I20250904 01:59:20.083060 21129 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.086864 21129 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3765bd467a4f4ceea91501064e04a61a, State: Initialized, Role: FOLLOWER
I20250904 01:59:20.087736 21129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.091152 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:20.092526 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:20.092963 21129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: configured and running, proceeding with master startup.
I20250904 01:59:20.093117 21104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:20.093974 21104 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036c187ff08844b1b3014de468c222a9 in term 2.
I20250904 01:59:20.095762 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a; no voters: c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:20.098155 21134 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 01:59:20.098718 21134 raft_consensus.cc:491] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2218317dfa74e5fa78821de6a65878f)
I20250904 01:59:20.099179 21134 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:59:20.108980 21134 raft_consensus.cc:513] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.112175 21104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a"
I20250904 01:59:20.112380 20384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "036c187ff08844b1b3014de468c222a9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c2218317dfa74e5fa78821de6a65878f"
I20250904 01:59:20.112692 21104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:59:20.116475 20305 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 FOLLOWER]: Rejecting Update request from peer c2218317dfa74e5fa78821de6a65878f for earlier term 2. Current term is 3. Ops: []
I20250904 01:59:20.118351 20984 consensus_queue.cc:1046] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 01:59:20.121119 21134 leader_election.cc:290] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 election: Requested vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), c2218317dfa74e5fa78821de6a65878f (127.16.112.188:36653)
I20250904 01:59:20.122882 21104 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036c187ff08844b1b3014de468c222a9 in term 3.
I20250904 01:59:20.123847 20282 leader_election.cc:304] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 036c187ff08844b1b3014de468c222a9, 3765bd467a4f4ceea91501064e04a61a; no voters: c2218317dfa74e5fa78821de6a65878f
I20250904 01:59:20.125072 21134 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 01:59:20.132131 21134 raft_consensus.cc:695] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 LEADER]: Becoming Leader. State: Replica: 036c187ff08844b1b3014de468c222a9, State: Running, Role: LEADER
I20250904 01:59:20.132879 21145 raft_consensus.cc:3053] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 LEADER]: Stepping down as leader of term 2
I20250904 01:59:20.133261 21145 raft_consensus.cc:738] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c2218317dfa74e5fa78821de6a65878f, State: Running, Role: LEADER
I20250904 01:59:20.133108 21134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.134173 21145 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.136430 21145 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 2 FOLLOWER]: Advancing to term 3
I20250904 01:59:20.148838 21146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036c187ff08844b1b3014de468c222a9. Latest consensus state: current_term: 3 leader_uuid: "036c187ff08844b1b3014de468c222a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:20.149427 21146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:20.152433 21151 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 01:59:20.158144 21151 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:59:20.161976 21151 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:20.162213 21151 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:59:20.166572 21151 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:59:20.170223 21151 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Loaded TSK: 0
I20250904 01:59:20.171485 21151 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:59:20.175164 21151 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:59:20.178767 21151 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:59:20.194932 21157 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:20.195204 21157 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: loading cluster ID for follower catalog manager: success
I20250904 01:59:20.204350 21157 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: acquiring CA information for follower catalog manager: success
I20250904 01:59:20.210480 21157 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250904 01:59:20.262198 20775 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:20.267537 21093 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d38c47cca76d44c6b7f52e18d380ae16" instance_seqno: 1756951150031790) as {username='slave'} at 127.16.112.130:56251; Asking this server to re-register.
I20250904 01:59:20.268972 20775 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:20.272215 21093 ts_manager.cc:194] Registered new tserver with Master: d38c47cca76d44c6b7f52e18d380ae16 (127.16.112.130:45285)
I20250904 01:59:20.288702 20633 heartbeater.cc:344] Connected to a master server at 127.16.112.190:37921
I20250904 01:59:20.291606 21093 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "49c95031b549406f9b2cf96a060d90a7" instance_seqno: 1756951147467597) as {username='slave'} at 127.16.112.129:41459; Asking this server to re-register.
I20250904 01:59:20.293008 20633 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:20.295859 21093 ts_manager.cc:194] Registered new tserver with Master: 49c95031b549406f9b2cf96a060d90a7 (127.16.112.129:36551)
I20250904 01:59:20.533157 20919 heartbeater.cc:499] Master 127.16.112.190:37921 was elected leader, sending a full tablet report...
I20250904 01:59:20.535952 21093 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c6a9423f201e49aaa10da3614d7169a2" instance_seqno: 1756951152291604) as {username='slave'} at 127.16.112.131:44265; Asking this server to re-register.
I20250904 01:59:20.538326 20919 heartbeater.cc:461] Registering TS with master...
I20250904 01:59:20.541145 21093 ts_manager.cc:194] Registered new tserver with Master: c6a9423f201e49aaa10da3614d7169a2 (127.16.112.131:40447)
I20250904 01:59:20.584225 21145 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 01:59:20.584683 21145 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.587013 21145 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:20.600404 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9" is_pre_election: true
I20250904 01:59:20.612564 21104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a" is_pre_election: true
I20250904 01:59:20.613255 21104 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2218317dfa74e5fa78821de6a65878f in term 3.
I20250904 01:59:20.614437 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f; no voters: 036c187ff08844b1b3014de468c222a9
I20250904 01:59:20.615489 21145 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 01:59:20.615805 21145 raft_consensus.cc:491] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 01:59:20.616137 21145 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:59:20.642459 21145 raft_consensus.cc:513] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.643674 20384 raft_consensus.cc:1238] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 4 FOLLOWER]: Rejecting Update request from peer 036c187ff08844b1b3014de468c222a9 for earlier term 3. Current term is 4. Ops: []
I20250904 01:59:20.647466 21146 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 01:59:20.648314 21134 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 LEADER]: Stepping down as leader of term 3
I20250904 01:59:20.648653 21134 raft_consensus.cc:738] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 036c187ff08844b1b3014de468c222a9, State: Running, Role: LEADER
I20250904 01:59:20.649333 21104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3765bd467a4f4ceea91501064e04a61a"
I20250904 01:59:20.649422 21134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.649920 21104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:59:20.650648 21134 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 3 FOLLOWER]: Advancing to term 4
I20250904 01:59:20.656400 21104 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2218317dfa74e5fa78821de6a65878f in term 4.
I20250904 01:59:20.657303 21145 leader_election.cc:290] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 4 election: Requested vote from peers 3765bd467a4f4ceea91501064e04a61a (127.16.112.190:37921), 036c187ff08844b1b3014de468c222a9 (127.16.112.189:33413)
I20250904 01:59:20.657429 20361 leader_election.cc:304] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3765bd467a4f4ceea91501064e04a61a, c2218317dfa74e5fa78821de6a65878f; no voters:
I20250904 01:59:20.658145 21145 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 4 FOLLOWER]: Leader election won for term 4
I20250904 01:59:20.659144 20305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2218317dfa74e5fa78821de6a65878f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "036c187ff08844b1b3014de468c222a9"
I20250904 01:59:20.660315 20305 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c2218317dfa74e5fa78821de6a65878f for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250904 01:59:20.666643 21145 raft_consensus.cc:695] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [term 4 LEADER]: Becoming Leader. State: Replica: c2218317dfa74e5fa78821de6a65878f, State: Running, Role: LEADER
I20250904 01:59:20.667410 21145 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } }
I20250904 01:59:20.672920 20375 catalog_manager.cc:2149] created HMS catalog entry for table [id=73f1917caebc4dac8ebc3c43e92ea14e]
W20250904 01:59:20.675876 20375 catalog_manager.cc:2177] Service unavailable: an error occurred while writing to the sys-catalog: leader is not yet ready
I20250904 01:59:20.677646 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2218317dfa74e5fa78821de6a65878f. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:20.678332 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:20.705533 20774 heartbeater.cc:499] Master 127.16.112.189:33413 was elected leader, sending a full tablet report...
I20250904 01:59:20.730948 20628 heartbeater.cc:499] Master 127.16.112.189:33413 was elected leader, sending a full tablet report...
I20250904 01:59:20.959336 20918 heartbeater.cc:499] Master 127.16.112.189:33413 was elected leader, sending a full tablet report...
I20250904 01:59:21.075150 21104 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [term 4 FOLLOWER]: Refusing update from remote peer c2218317dfa74e5fa78821de6a65878f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250904 01:59:21.076745 21167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 01:59:21.103114 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2218317dfa74e5fa78821de6a65878f. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:21.103966 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:21.114742 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:21.115429 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3765bd467a4f4ceea91501064e04a61a [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:21.116858 20305 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 4 FOLLOWER]: Refusing update from remote peer c2218317dfa74e5fa78821de6a65878f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250904 01:59:21.117203 20305 pending_rounds.cc:85] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Aborting all ops after (but not including) 5
I20250904 01:59:21.117489 20305 pending_rounds.cc:107] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250904 01:59:21.118536 20305 raft_consensus.cc:2887] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250904 01:59:21.120239 21167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 01:59:21.129158 21145 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:21.129925 21145 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f [sys.catalog]: This master's current role is: LEADER
I20250904 01:59:21.130108 21146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:21.130760 21146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:21.131116 21134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2218317dfa74e5fa78821de6a65878f. Latest consensus state: current_term: 4 leader_uuid: "c2218317dfa74e5fa78821de6a65878f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3765bd467a4f4ceea91501064e04a61a" member_type: VOTER last_known_addr { host: "127.16.112.190" port: 37921 } } peers { permanent_uuid: "036c187ff08844b1b3014de468c222a9" member_type: VOTER last_known_addr { host: "127.16.112.189" port: 33413 } } peers { permanent_uuid: "c2218317dfa74e5fa78821de6a65878f" member_type: VOTER last_known_addr { host: "127.16.112.188" port: 36653 } } }
I20250904 01:59:21.131769 21134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036c187ff08844b1b3014de468c222a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250904 01:59:21.296798 20375 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=73f1917caebc4dac8ebc3c43e92ea14e]
I20250904 01:59:21.298020 20375 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41628 (request call id 2) took 1259 ms. Trace:
I20250904 01:59:21.298372 20375 rpcz_store.cc:276] 0904 01:59:20.038434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0904 01:59:20.039506 (+ 1072us) service_pool.cc:225] Handling call
0904 01:59:20.054294 (+ 14788us) catalog_manager.cc:2076] Acquired catalog manager lock
0904 01:59:20.061996 (+ 7702us) catalog_manager.cc:2122] Created new table and tablet info
0904 01:59:20.672881 (+610885us) catalog_manager.cc:2148] Created new table in HMS catalog
0904 01:59:20.675563 (+ 2682us) write_op.cc:489] Releasing partition, row and schema locks
0904 01:59:20.675600 (+ 37us) write_op.cc:454] Released schema lock
0904 01:59:20.676139 (+ 539us) catalog_manager.cc:2156] Rolling back HMS table creation
0904 01:59:21.297930 (+621791us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0904 01:59:20.675253 (+ 0us) op_driver.cc:389] HandleFailure(Service unavailable: leader is not yet ready)
0904 01:59:20.675319 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0904 01:59:20.675352 (+ 33us) write_op.cc:454] Released schema lock
0904 01:59:20.675392 (+ 40us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":174,"Hive Metastore.run_cpu_time_us":1171,"Hive Metastore.run_wall_time_us":1229663,"child_traces":[["op",{}]]}
I20250904 01:59:21.300715 21169 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master_failover-itest.cc:471: Failure
Failed
Bad status: Service unavailable: Error creating table default.table_0 on the master: an error occurred while writing to the sys-catalog: leader is not yet ready
I20250904 01:59:21.303598 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20486
I20250904 01:59:21.307183 21169 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 01:59:21.309751 21169 catalog_manager.cc:1261] Loaded cluster ID: a6e1ac00b598454f96fde9d43704920d
I20250904 01:59:21.309953 21169 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 01:59:21.312950 21169 catalog_manager.cc:1506] Loading token signing keys...
I20250904 01:59:21.315476 21169 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c2218317dfa74e5fa78821de6a65878f: Loaded TSK: 0
I20250904 01:59:21.316211 21169 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 01:59:21.318665 21169 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250904 01:59:21.321573 21169 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250904 01:59:21.336122 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20632
I20250904 01:59:21.368146 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20781
I20250904 01:59:21.401335 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 21047
I20250904 01:59:21.427706 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20258
I20250904 01:59:21.453505 16834 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task_fQbLu/build/tsan/bin/kudu with pid 20332
Shutting down hive metastore.
2025-09-04T01:59:21Z chronyd exiting
I20250904 01:59:21.870185 16834 test_util.cc:183] -----------------------------------------------
I20250904 01:59:21.870383 16834 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task_fQbLu/test-tmp/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1756951033459069-16834-0
[ FAILED ] HmsConfigurations/MasterFailoverTest.TestMasterPermanentFailure/1, where GetParam() = 2 (37722 ms)
[----------] 4 tests from HmsConfigurations/MasterFailoverTest (128262 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (128262 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] HmsConfigurations/MasterFailoverTest.TestCreateTableSync/1
[ SKIPPED ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1
[ FAILED ] 1 test, listed below:
[ FAILED ] HmsConfigurations/MasterFailoverTest.TestMasterPermanentFailure/1, where GetParam() = 2
1 FAILED TEST